AI Inference: 10x Faster Models & Self-Optimization

In a recent episode of the Latent Space podcast, hosts and guests delved into the intricate world of AI inference, exploring how large language models (LLMs) are being optimized for speed and efficiency. Philip Kiely, author of "Inference Engineering," and Ali Taha, Head of Model Performance at Baseten, shared insights into the techniques and challenges involved in getting AI models to perform at their peak.

The Power of Custom Kernels and Speculative Decoding

The conversation highlighted the remarkable capability of LLMs, such as GLM-5.2, to not only generate text but also to write optimized GPU kernels. This process involves a loop where the model analyzes profiling traces, identifies bottlenecks in its inference engine, writes new kernels, and then re-evaluates. "Some of the GPU kernels that we run GLM-5.2 with in our inference engine is written by GLM-5.2," explained one of the speakers, showcasing the self-optimizing potential of AI.

Speculative decoding emerged as a key technique for accelerating LLM inference. This method involves attaching a smaller, faster model that predicts several tokens ahead. The larger, original model then verifies these predictions in a single forward pass. Philip Kiely explained, "We attach like this little like kind of parasite like this layer that goes on top of the model. And this model just has to predict it does three very fast ultra reagressive over passes and it will predict like you know three certain tokens and then you do one forward stage over the entire original model in order to see if those predictions were correct or not and then you accept them or you reject them." The effectiveness of this approach hinges on the specificity of the task; a speculative model trained on, for example, summarizing Harry Potter books, would yield higher acceptance rates than a general-purpose one.

Navigating Long Queries and Tool Calling Challenges

When handling long queries, such as those with 200,000 tokens, the process prioritizes caching. "The first thing that I'm going to ask is, have you sent me this query before or at least part of it?" noted Ali Taha. "Ideally some cached input already there so that we can skip prefill on at least part of these 200,000 tokens." This strategy is particularly useful for applications like coding or multi-turn agents where context is expected to be preserved.

The discussion also touched upon the complexities of tool calling. While the core idea is to generate structured output like JSON, the challenge lies in training models for specific tool-calling scenarios. "The challenge with tool calling more and more seems to be that the companies want certain tool calling which is a very sensitive thing to train," Taha observed. He elaborated on the difficulties arising from JSON output formatting, where improper closure can lead to hallucinations. To address this, the team has implemented a state machine approach to constrain the output format, ensuring structured and reliable results.

The Engineering Behind Model Support and Open Source Collaboration

Supporting new model releases, like GLM-5.2 or Mixtral-8x7B, involves significant engineering effort beyond just making them functional. This includes redoing quantization work, such as converting models to NVFP4 for optimal performance, and ensuring that these optimizations don't degrade the model's intelligence. "We have to perform that quantization and calibrate the quantization to make sure that we're not causing any kind of regression in the model's intelligence," explained one of the speakers.

The power of open-source AI was a recurring theme. The ability to combine components from different models, such as "Kimi vision encoder onto GLM-5.2," exemplifies how the community can collaboratively build more capable systems. This modular approach allows for rapid innovation and the creation of models that surpass the individual capabilities of their constituent parts.

Optimizing for Performance: Beyond Benchmarks

The conversation highlighted the continuous drive for inference speed improvements, with providers often engaging in an "inference speed race." While tokens per second (TPS) is a common metric, the speakers emphasized the importance of understanding the nuances, such as inter-token latency. They noted that significant gains, often multiples of current performance, are achievable through stacked optimizations like quantization, speculative decoding, and efficient KV caching.

The discussion also touched upon the challenges of benchmarking and the non-deterministic nature of models, which can sometimes be influenced by hardware and subtle software issues. Ultimately, the goal in inference engineering is to achieve faster inference while maintaining fidelity to the original model's performance, ensuring that users cannot discern a difference between the optimized API and the original model.