DigitalOcean: Model Routing Beats Benchmarks

In the rapidly evolving world of AI, the pursuit of the 'best' model often leads teams to chase benchmark scores. However, Archana Kamath, VP of Engineering for Inference Engine and AI Infrastructure at DigitalOcean, argues this is a flawed instinct. Speaking at the AI Engineer World's Fair, Kamath, alongside Tyler Gillam, who built parts of DigitalOcean's inference router, explained why preferences, not just benchmarks, should dictate model selection.

The Three Forces Breaking the One-Model Habit

Kamath highlighted three key reasons why the traditional approach of relying on a single, top-performing model is no longer sufficient:

• Cost: Inference bills are exploding, with major companies like Walmart, Uber, and Microsoft actively capping usage to control costs.
• Fit: A single, powerful model is often overkill for many tasks. Developers end up paying premium rates for work that smaller, more specialized models can handle effectively.
• Risk: Relying on a single model introduces significant risk. If that model experiences downtime or degradation, an entire product can be compromised with no failover.

Kamath emphasized that "Model orchestration is actually the new FinOps." She noted that while cloud cost optimization took years to mature, AI model cost optimization is arriving in months, driven by these critical factors.

The Right Model Depends on the Request

There isn't a single 'best' model for every job. The optimal choice depends on a combination of factors:

• Task: What is the model actually trying to achieve? For classification and labeling, a small open model might suffice, offering significant cost savings.
• System Prompts & Tools: The scaffolding around the model and how it's prompted influences its performance.
• Cost Limits: The budget available for a specific request is a crucial consideration.
• Latency Needs: Different use cases have varying latency requirements. For inline code completion, fast routing is essential, potentially requiring a faster, larger model.
• End-User Preference: Ultimately, the user's experience and expectations drive the model selection.

For tasks like code generation and bug fixing, a mid-weight open model can provide good cost optimizations. However, for accuracy-critical tasks such as code review and security, a more advanced, 'frontier' model might be necessary.

DigitalOcean's Purpose-Built Router

Tyler Gillam introduced DigitalOcean's inference router, explaining its different approach:

"Many builders have tried auto-routing before, but the problem was that it feels like a black box. The router makes a choice and if that choice results in poor performance, you really have no way of improving it. We built ours differently at the architecture level."

The system utilizes an open proxy plan and a purpose-built routing model, both open source, ensuring no vendor lock-in. Users describe what matters for their workload, cost, latency, quality, preferred models, or hard rules, and the router uses this context to select the appropriate model per request. This specialized routing model is fast, operating in under 200 milliseconds, and is free of charge.

"Routing decision and under 200 milliseconds per request," Kamath added. "It runs on a custom mixture of experts model purpose-built for routing. Zero application code changes needed from you to get it to adopt and it's free and included so you do not have to roll out your own router."

Live Demo and Key Features

Gillam then demonstrated the router's capabilities, showcasing its UI for configuration and evaluation. He highlighted presets for common tasks like software engineering and general writing, as well as the ability to create custom routers.

The demo illustrated how the router dynamically selects different models based on the task, leading to significant cost and latency improvements compared to using a single, high-end model like Opus for all requests. For instance, a code generation task was routed to a faster, cheaper model, while a more complex task might utilize a different, more powerful option.

"The difference is simple. This is routing you can customize, evaluate and improve without vendor lock in."

The presentation also touched on the importance of validation through evaluations, emphasizing that teams should trust their own benchmarks over external leaderboards. DigitalOcean's approach facilitates this by allowing users to define their preferences and validate the router's performance with their own data.

The Foundation for Future AI Development

Kamath concluded by positioning routing as a foundational layer for AI development, with further enhancements like caching and personalization building upon it. "Routing is the foundation, not the destination," she stated. "The more you route and evaluate, the better the router does for your workload."

DigitalOcean's commitment to an open-source, preference-driven approach aims to empower teams to build more efficiently and cost-effectively, ensuring they can always select the 'right' model for every specific request.