Artificial Intelligence
Preferred on Google
Google News

DigitalOcean: Model Routing Beats Benchmarks

DigitalOcean's Archana Kamath and Tyler Gillam discuss model routing, arguing that preferences like cost and latency should dictate LLM choices over benchmarks, and showcase their open-source inference router.

9 min read
DigitalOcean's Archana Kamath and Tyler Gillam presenting on model routing at AI Engineer World's Fair.
Archana Kamath and Tyler Gillam of DigitalOcean discuss preferences over benchmarks for AI model routing.· AI Engineer

Visual TL;DR. One-Model Habit Flawed driven by 3 Forces Break Habit. 3 Forces Break Habit requires Right Model Per Request. One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router with Router Key Features. DO Purpose-Built Router achieves Beats Benchmarks. Beats Benchmarks enables Future AI Foundation.

  1. One-Model Habit Flawed: chasing benchmarks for LLM selection often leads to suboptimal choices
  2. 3 Forces Break Habit: exploding inference costs, poor model fit, and single-model risk
  3. Right Model Per Request: preferences like cost and latency should dictate LLM choices
  4. DO Purpose-Built Router: DigitalOcean's open-source inference router showcased at AI Engineer World's Fair
  5. Router Key Features: enables dynamic model selection based on user-defined preferences
  6. Beats Benchmarks: model routing approach outperforms traditional benchmark-driven selection
  7. Future AI Foundation: lays groundwork for more efficient and adaptable AI development
Visual TL;DR
Visual TL;DR, startuphub.ai One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router achieves Beats Benchmarks leads to implemented by achieves One-Model Habit Flawed Right Model Per Request DO Purpose-Built Router Beats Benchmarks From startuphub.ai · The publishers behind this format
Visual TL;DR, startuphub.ai One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router achieves Beats Benchmarks leads to implemented by achieves One-Model HabitFlawed Right Model PerRequest DO Purpose-BuiltRouter Beats Benchmarks From startuphub.ai · The publishers behind this format
Visual TL;DR, startuphub.ai One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router achieves Beats Benchmarks leads to implemented by achieves One-Model Habit Flawed chasing benchmarks for LLM selection oftenleads to suboptimal choices Right Model Per Request preferences like cost and latency shoulddictate LLM choices DO Purpose-Built Router DigitalOcean's open-source inferencerouter showcased at AI Engineer World'sFair Beats Benchmarks model routing approach outperformstraditional benchmark-driven selection From startuphub.ai · The publishers behind this format
Visual TL;DR, startuphub.ai One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router achieves Beats Benchmarks leads to implemented by achieves One-Model HabitFlawed chasing benchmarksfor LLM selectionoften leads to… Right Model PerRequest preferences likecost and latencyshould dictate LLM… DO Purpose-BuiltRouter DigitalOcean'sopen-sourceinference router… Beats Benchmarks model routingapproachoutperforms… From startuphub.ai · The publishers behind this format
Visual TL;DR, startuphub.ai One-Model Habit Flawed driven by 3 Forces Break Habit. 3 Forces Break Habit requires Right Model Per Request. One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router with Router Key Features. DO Purpose-Built Router achieves Beats Benchmarks. Beats Benchmarks enables Future AI Foundation driven by requires leads to implemented by with achieves enables One-Model Habit Flawed chasing benchmarks for LLM selection oftenleads to suboptimal choices 3 Forces Break Habit exploding inference costs, poor model fit,and single-model risk Right Model Per Request preferences like cost and latency shoulddictate LLM choices DO Purpose-Built Router DigitalOcean's open-source inferencerouter showcased at AI Engineer World'sFair Router Key Features enables dynamic model selection based onuser-defined preferences Beats Benchmarks model routing approach outperformstraditional benchmark-driven selection Future AI Foundation lays groundwork for more efficient andadaptable AI development From startuphub.ai · The publishers behind this format
Visual TL;DR, startuphub.ai One-Model Habit Flawed driven by 3 Forces Break Habit. 3 Forces Break Habit requires Right Model Per Request. One-Model Habit Flawed leads to Right Model Per Request. Right Model Per Request implemented by DO Purpose-Built Router. DO Purpose-Built Router with Router Key Features. DO Purpose-Built Router achieves Beats Benchmarks. Beats Benchmarks enables Future AI Foundation driven by requires leads to implemented by with achieves enables One-Model HabitFlawed chasing benchmarksfor LLM selectionoften leads to… 3 Forces BreakHabit exploding inferencecosts, poor modelfit, and… Right Model PerRequest preferences likecost and latencyshould dictate LLM… DO Purpose-BuiltRouter DigitalOcean'sopen-sourceinference router… Router KeyFeatures enables dynamicmodel selectionbased on… Beats Benchmarks model routingapproachoutperforms… Future AIFoundation lays groundwork formore efficient andadaptable AI… From startuphub.ai · The publishers behind this format

In the rapidly evolving world of AI, the pursuit of the 'best' model often leads teams to chase benchmark scores. However, Archana Kamath, VP of Engineering for Inference Engine and AI Infrastructure at DigitalOcean, argues this is a flawed instinct. Speaking at the AI Engineer World's Fair, Kamath, alongside Tyler Gillam, who built parts of DigitalOcean's inference router, explained why preferences, not just benchmarks, should dictate model selection.

DigitalOcean: Model Routing Beats Benchmarks - AI Engineer
DigitalOcean: Model Routing Beats Benchmarks — from AI Engineer

The Three Forces Breaking the One-Model Habit

Kamath highlighted three key reasons why the traditional approach of relying on a single, top-performing model is no longer sufficient:

  • Cost: Inference bills are exploding, with major companies like Walmart, Uber, and Microsoft actively capping usage to control costs.
  • Fit: A single, powerful model is often overkill for many tasks. Developers end up paying premium rates for work that smaller, more specialized models can handle effectively.
  • Risk: Relying on a single model introduces significant risk. If that model experiences downtime or degradation, an entire product can be compromised with no failover.

Kamath emphasized that "Model orchestration is actually the new FinOps." She noted that while cloud cost optimization took years to mature, AI model cost optimization is arriving in months, driven by these critical factors.

The Right Model Depends on the Request

There isn't a single 'best' model for every job. The optimal choice depends on a combination of factors:

  • Task: What is the model actually trying to achieve? For classification and labeling, a small open model might suffice, offering significant cost savings.
  • System Prompts & Tools: The scaffolding around the model and how it's prompted influences its performance.
  • Cost Limits: The budget available for a specific request is a crucial consideration.
  • Latency Needs: Different use cases have varying latency requirements. For inline code completion, fast routing is essential, potentially requiring a faster, larger model.
  • End-User Preference: Ultimately, the user's experience and expectations drive the model selection.

For tasks like code generation and bug fixing, a mid-weight open model can provide good cost optimizations. However, for accuracy-critical tasks such as code review and security, a more advanced, 'frontier' model might be necessary.

DigitalOcean's Purpose-Built Router

Tyler Gillam introduced DigitalOcean's inference router, explaining its different approach:

"Many builders have tried auto-routing before, but the problem was that it feels like a black box. The router makes a choice and if that choice results in poor performance, you really have no way of improving it. We built ours differently at the architecture level."

The system utilizes an open proxy plan and a purpose-built routing model, both open source, ensuring no vendor lock-in. Users describe what matters for their workload, cost, latency, quality, preferred models, or hard rules, and the router uses this context to select the appropriate model per request. This specialized routing model is fast, operating in under 200 milliseconds, and is free of charge.

"Routing decision and under 200 milliseconds per request," Kamath added. "It runs on a custom mixture of experts model purpose-built for routing. Zero application code changes needed from you to get it to adopt and it's free and included so you do not have to roll out your own router."

Live Demo and Key Features

Gillam then demonstrated the router's capabilities, showcasing its UI for configuration and evaluation. He highlighted presets for common tasks like software engineering and general writing, as well as the ability to create custom routers.

The demo illustrated how the router dynamically selects different models based on the task, leading to significant cost and latency improvements compared to using a single, high-end model like Opus for all requests. For instance, a code generation task was routed to a faster, cheaper model, while a more complex task might utilize a different, more powerful option.

"The difference is simple. This is routing you can customize, evaluate and improve without vendor lock in."

The presentation also touched on the importance of validation through evaluations, emphasizing that teams should trust their own benchmarks over external leaderboards. DigitalOcean's approach facilitates this by allowing users to define their preferences and validate the router's performance with their own data.

The Foundation for Future AI Development

Kamath concluded by positioning routing as a foundational layer for AI development, with further enhancements like caching and personalization building upon it. "Routing is the foundation, not the destination," she stated. "The more you route and evaluate, the better the router does for your workload."

DigitalOcean's commitment to an open-source, preference-driven approach aims to empower teams to build more efficiently and cost-effectively, ensuring they can always select the 'right' model for every specific request.

© 2026 StartupHub.ai. All rights reserved. Do not enter, scrape, copy, reproduce, or republish this article in whole or in part. Use as input to AI training, fine-tuning, retrieval-augmented generation, or any machine-learning system is prohibited without written license. Substantially-similar derivative works will be pursued to the fullest extent of applicable copyright, database, and computer-misuse laws. See our terms.
#DigitalOcean#AI Engineer World's Fair#Model Routing#Inference Engine#LLM