AI Model Routing: NVIDIA, Cognition, OpenRouter Panel

The evolving landscape of AI deployment, particularly for local inference, necessitates efficient strategies for model selection and utilization. In a panel discussion titled "The State of Model Routing," industry leaders from NVIDIA, Cognition, and OpenRouter convened to explore the challenges and advancements in this critical area.

The Multi-Model Reality

Nader Khalil, Director of Developer Tech at NVIDIA, opened the discussion by highlighting the industry's shift towards a "multi-model world." He emphasized that companies deploying AI locally are increasingly encountering a scenario where multiple models are needed to handle diverse tasks. This reality, he noted, is why NVIDIA releases its NeMo models with open datasets and weights, enabling customization and flexibility.

Cognition's Approach to Model Routing

Walden Yan, Co-founder of Cognition, shared insights into the company's work on "Devon AI," an AI software engineer. He explained that a key challenge for their customers is determining the return on investment (ROI) for their AI models and identifying which tasks can be delegated to more cost-efficient models versus relying on expensive, frontier models. This has led Cognition to focus heavily on multi-model routing.

Cognition recently launched "Fusion," a model router designed to improve performance beyond that of single frontier models. Yan clarified that this doesn't mean Fusion models are inherently superior in every aspect, but rather that smarter models can excel at delegating work. The philosophy behind Fusion is to avoid routing users to a less capable model that might force them to switch back to a more expensive one. Their system aims to reduce the cost of frontier-level intelligence by up to 40% by allowing powerful models to handle planning and complex decision-making, while delegating execution tasks to cheaper, specialized models.

Yan elaborated on an unintuitive benefit: by delegating tasks to smaller models, they can be directed with more depth and intensity, potentially leading to more comprehensive results than a single model could achieve alone. This creates a trade-off where the system is both more cost-efficient and more comprehensive.

The Nuances of Model Selection

Tanay Varshney, a Principal Engineer at NVIDIA, added that model routing involves understanding the "jagged capabilities" of different models. He stressed that a model scoring high on one benchmark, like coding, doesn't guarantee superior performance across all tasks, such as data visualization. The strengths and weaknesses of models are largely determined by their training corpora.

"Routing is a task of intimately understanding the behavioral and strengths and weaknesses of different models and then applying them thusly," Varshney stated. He encouraged thinking of models as having complementary strengths rather than a single "model to rule them all."

Orchestrating Swarms of Agents

Alex Atallah, Cofounder & CEO of OpenRouter, introduced the idea of using a "swarm of agents" to accomplish tasks, where the routing between these agents becomes a complex problem in itself. He used the example of scientific discovery, which often involves many subdomains and specialized models.

Varshney expanded on this, suggesting that understanding how models fail on different subtasks can inform orchestration strategies. He posited that by leveraging these complementary strengths, significant improvements in accuracy, potentially up to 10%, can be achieved, depending on the model pool and task at hand.

Managing Context and Cost

A key challenge discussed was managing context sharing between models, especially when dealing with multiple agents. The concern is that passing the entire trace of a smaller agent's work back to a larger model could be inefficient and costly. The solution lies in fine-tuning the ability of smaller models to present context effectively, potentially through techniques like context compaction, which have been studied for long-running agents.

Yan noted that running multiple models together can easily become more expensive if not managed carefully, as redundant file reads or token generation can multiply costs. The focus is on ensuring that most context only goes to the necessary model while still providing high-level thinking and file references back to the orchestrator.

The Future of Model Routing

The panel agreed that the field of model routing is still nascent. Yan expressed hope that current techniques, like those in Devon Fusion, will be seen as "legacy ideas" in a year, with better methods for inter-model routing emerging. He also highlighted the importance of co-designing models with orchestration in mind, rather than simply orchestrating existing models.

NVIDIA's approach involves technologies like "FlexRun," which allows for distilling a main model into smaller footprints and switching between models based on the task at hand. This flexibility, especially for open models where data recipes are accessible, allows for decisions based on whether a query is "in-distribution" or "out-of-distribution" for a particular model.

Balancing Performance and Cost

The discussion touched upon the potential for smaller models to outperform larger ones in specific, in-domain tasks, but cautioned against using overly small models for out-of-domain tasks, where they might increase costs due to thrashing and inefficient tool use. The example of Opus outperforming Haiku on a terminal bench task at a lower cost, despite Haiku being cheaper per token, illustrated this point.

The conversation also delved into the concept of "hallucination probes" as a proxy for a model's confusion, suggesting that a higher tendency to hallucinate might indicate a model is "lost in its thinking."

Prompt Engineering and Collaboration

The portability of prompts across different model architectures was raised as a challenge. Yan emphasized that the value of agent companies like Devon lies in discovering and recovering from "doom loops" across industries, which directly influences prompt engineering. He suggested that prompts are integral to the startup building process and can be iterated upon by observing traces and adjusting accordingly.

When asked about prompt tuning, Yan expressed less confidence in low-level mechanical prompt tuning frameworks compared to simply instructing a smart model with context and decisions, and then having an agent update prompts based on performance feedback.

The Evolving Role of Orchestration

The panel concluded by discussing whether routers will become standalone products or integrated plumbing. The consensus was that models themselves are becoming more collaborative and better at delegation, while harnesses need to be designed with multi-model workflows in mind. The future likely involves a symbiotic relationship between sophisticated models and intelligent orchestration systems.

The discussion highlighted the dynamic nature of AI development, with model routing and orchestration emerging as key areas for innovation, driving both performance improvements and cost efficiencies across the AI landscape.