#Inference Optimization
8 articles with this tag
Test-Time Distillation Nearly Doubles Model Performance
New research shows stronger AI models can guide weaker ones at inference time, nearly doubling performance without retraining through 'scaffolding'.
Matryoshka: Nested LMs for Efficiency
The Matryoshka training framework nests language model sub-models, drastically cutting compute costs and enhancing speculative decoding throughput while maintaining performance parity.
MiniMax M3: Open Source AI Model Deep Dive
Dan from Together AI and Olive from MiniMax discuss the open-sourcing of the M3 multimodal AI model, its capabilities, and the infrastructure behind scaling AI.
LOCKS Unlocks LLM Long Context Efficiency
LOCKS revolutionizes long-context LLMs by approximating KV cache with spectral summaries, achieving near full-quality inference while drastically cutting latency and computation.
FlashRT: Execution State for Latency-First AI
FlashRT revolutionizes on-device AI serving with execution-state capsules, enabling sub-millisecond state restoration and significant TTFT speedups for latency-critical applications.
Cloudflare Unweights LLMs by 22%
Cloudflare's 'Unweight' system slashes LLM model sizes by up to 22% using lossless compression, enhancing inference speed and efficiency.
LLM Adaptation Without Retraining
In-Place Test-Time Training enables LLMs to adapt to new data at inference without retraining, enhancing performance and paving the way for continual learning.
GPT-OSS-Puzzle-88B: Faster AI, Same Brains
GPT-OSS-Puzzle-88B offers substantial inference speedups for large language models without sacrificing accuracy, utilizing techniques like MoE pruning and window attention.