#Quantization
7 articles with this tag
AI Inference: 10x Faster Models & Self-Optimization
Philip Kiely and Ali Taha of Baseten discuss AI inference, LLM optimization, speculative decoding, and the engineering behind cutting-edge AI models.
Google AI Edge: Tiny LMs Powering Robotics & Devices
Google's Cormac Brick discusses the state-of-the-art in tiny AI models, their applications in robotics and edge devices, and the importance of fine-tuning for broad accessibility.
TurboQuant: Supercharging AI Agent Retrieval with Compression
Shashi Jagtap of Superagentic AI introduces TurboQuant, a method to compress AI agent memory and embeddings, reducing usage by 5-8x with no quality loss.
Nvidia's Ziv Ilan on Faster Diffusion Models
Nvidia's Ziv Ilan explains how to reduce diffusion model latency using quantization, caching, and distillation, plus the new FastGen library.
AI Model Compression: Key to Efficient LLM Deployment
Cedric Clyburn of Redh explains how AI model compression, especially quantization, is crucial for efficient LLM deployment, reducing costs and improving performance.
Run LLMs Locally with Llama.cpp
Cedric Clyburn explains how Llama.cpp makes running large language models locally on consumer hardware possible, highlighting GGUF format and optimized kernels for efficiency and accessibility.
Edge AI Acceleration Gets Flexible
Researchers developed a novel FPGA-based accelerator that dynamically adjusts neural network precision at runtime, boosting inference speed for edge AI.