LLM Gateway Tradeoffs Twilio Learned in Prod

An LLM gateway looks simple until a provider outage turns "something went wrong" into your product, according to AI Engineer.

Kanish Manuja, principal engineer for conversational AI at Twilio (NYSE:TWLO), laid out the architecture tradeoffs in a production talk on why gateways break quietly.

He leads real-time voice and agentic workflow infrastructure at Twilio, where 300,000 customers depend on low-latency LLM calls.

What is an LLM gateway?

Manuja defined it as middleware between apps and model providers that handles routing, authentication, fallback, rate limits and governance.

At its core the gateway forces a four-way tradeoff between availability, latency, guardrails and cost, and you cannot maximize all four during a degradation.

The point of a good gateway is to give callers explicit levers to choose that tradeoff per use case.

Why do retries and circuit breakers fail for LLMs?

Single provider ceiling is your ceiling, and typical software retries with exponential backoff and jitter do not translate.

LLM calls are slow and expensive, so blind retries multiply tail latency and double your bill.

Tripping a circuit breaker while a healthy second provider sits idle wastes capacity you already paid for.

How should fallback actually work?

Per-request fallback tries provider A then provider B in sequence, and parallel dispatch is only for teams obsessed with latency because it doubles cost.

If the primary has failed for a window you remove it from the load balancer, cool it down for minutes, then probe it again.

Failure counters can live in memory on each instance or shared across the fleet, and fleetwide state gives faster failover while local state shifts with deployment size.

Why are fallbacks not transparent?

Even with OpenAI-compatible APIs, providers diverge on tool calling schemas, token limits and stop reasons, so untested fallbacks break in production.

A normalization layer in the gateway is required to make cross-provider fallback reliable.

Streaming makes it harder because once you start sending tokens from provider A you cannot mid-stream switch to provider B without showing that "something went wrong" message by design.

Where do teams under-provision?

Teams harden the primary provider and neglect the fallback, yet the fallback needs more headroom because it is the last line of defense.

StartupHub.ai data shows how crowded this control plane is, with our own score at 71/100 and verified $80M Series A in 2023, versus Perplexity AI at 72/100 and Google (NASDAQ:GOOGL) at 74/100, plus Lucidworks at 52/100, matey at 50/100 and Dante at 47/100.

The open market already reflects Manuja's pattern: LiteLLM, Portkey, Cloudflare AI Gateway and Vercel AI Gateway all implement router fallbacks and hedging as standard.

How should you measure latency?

Availability failures page you, but latency degrades silently and needs more attention.

Aggregate gateway latency is a lie when you mix embeddings under a second with chat at three seconds and reasoning jobs that wander, so track P99 per model per route.

Set timeouts per model class per route, because without them the gateway thinks a stuck request is happily served while users wait.

What makes reasoning models special?

Reasoning and router models are nondeterministic, and the same prompt can swing from 2 seconds to 60 seconds with P99 popping for no obvious reason.

Fix the reasoning level per route to make behavior deterministic where possible, and hedge the tail by firing a second request once you pass P90 of your latency budget.

How do guardrails change the tradeoff?

Guardrails for prompt injection, PII, toxicity and profanity are themselves unreliable services, so you must decide to fail open and serve or fail closed and block when they are down.

Your request should never be bound by guardrail timing, so give guardrails a strict time budget and keep the LLM as the rate-determining step.

Guardrails also deserve fallbacks, secondary checks and cached decisions just like model providers.

Where should guardrails run?

Pre-hooks on input are safest but add serial latency.

Parallel execution saves latency for structured outputs, but it does not work well with streaming.

Post-hooks are best for output monitoring and auditing.

What breaks inside the gateway itself?

Shared rate limits let one noisy tenant starve others, so segregate API keys per route per use case to the most granular level you can.

Under retry storms you cannot simply scale out, so support load shedding, bounded web server queues and traffic prioritization to protect critical use cases.

Should you centralize the gateway?

A central gateway for the whole company is a single point of failure, and most teams actually want centralized governance, not centralized traffic.

Decentralize the gateway deployments and centralize governance through plugins for cost tracking and rate limit management instead.

That split keeps availability up while preserving control, which is the entire point Manuja ended on: prevent one incident today rather than explaining one tomorrow.