Harshul Jain and Parv Khatri told AI Engineer that self-hosted models must learn to say no before GPU memory commits.
Their breaking case is 40 requests in two seconds against a ceiling of 8 concurrent requests, Jain and Khatri said. A coding agent fixing one failing test can trigger 14 file reads plus 10 test runs, or 40 to 50 model calls at once. A person sends one prompt and reads, an agent fires bursts with retries and pauses. That shift puts the decision at the front door.
OpenAI said its routing servers ran out of memory after heavy traffic and recovered by cutting intake, per the postmortem cited by Jain and Khatri. Anthropic said its Sonnet 4.5 tier slowed then errored for about 90 minutes for lack of compute, per the postmortem cited by Jain and Khatri. Neither team asked for a faster engine, their fixes were graceful load shedding and earlier detection of limits.
A yes is not a moment, it is a reservation.
Inference runs in two phases, prefill builds KV cache from the prompt and decode writes one token at a time, Jain explained. That memory stays reserved until finish or disconnect, so one long answer can block short requests behind it. The fix sits at ingress with three answers, admit with 200, defer with 503 and retry-after, or reject with 429 for caller overuse and 529 for pool overload. Agents should start timeouts at first token, close dead streams to free cache, and spread retries with jitter.
Self-hosted inference means running models locally instead of sending prompts to OpenAI or Anthropic servers, which leaves privacy and GPU budget teams with no provider cushion. Fixed gateway caps count requests but never price KV memory or tokens, so queues grow and clients time out. Jain, senior software engineer at Audible, and Khatri, senior software developer at SAP, argued cost-aware admission must read live engine state and prove itself in benchmarks. Without it, one agent loop turns 40 calls into 80.
