How Together AI Solves LLM Cold Starts

Traditional CPU autoscaling fails when applied to large language models. A GPU running at 60% capacity can hide a severe queue bottleneck, pushing time-to-first-token latency from 200 milliseconds to 15 seconds. To address these hardware quirks, Together AI introduced native metrics tailored for autoscaling endpoints for LLM inference on dedicated infrastructure.

LLM serving breaks the core assumptions of classic stateless web services. Hardware like the H100 and B200 accelerators from Nvidia (NASDAQ:NVDA) requires gigabytes of model weights to be fetched, loaded into VRAM, and warmed up before processing a single request. When sudden traffic spikes arrive, classic scaling infrastructure reacts too late.

The Latency Fallacy of Hardware Metrics

Standard CPU and memory metrics lie about generative workload pressure. A GPU metric often measures arithmetic intensity rather than queue depth. If an inference engine reaches its concurrency limit, performance does not degrade gracefully. Instead, requests backlog immediately.

Together AI addresses this by giving teams eight specialized metrics to drive their scaling loops. These include leading indicators like inflight_requests, trailing SLO indicators like ttft and e2e_latency, and cost-focused metrics like gpu_utilization and token_utilization. By default, the platform targets 8 in-flight requests per replica. That default provides a buffer before queueing damages latency.

The Brutal Math of GPU Cold Starts

Cold starts on dedicated hardware remain a massive operational bottleneck. Benchmarks on 1x H100 replicas show that booting a standard Qwen3.5-9B model takes 86 seconds from creation to ready status. Loading a custom fine-tune with 18GB of weights extends that process to 145 seconds. Routing the first token adds another 26 to 40 seconds.

Scaling an endpoint from 1 to 2 replicas takes roughly 2.5 minutes, while restarting a stopped deployment with warm weights takes 1 to 2 minutes. Because cold starts take minutes rather than milliseconds, scaling out after a spike hits guarantees degraded user performance.

What This Means for Founders and AI Builders

For early-stage startups managing tight infrastructure budgets, setting min_replicas to zero seems like an easy way to save capital. But on dedicated GPU setups, scale-to-zero without auto-wake forces cold start penalties onto users. Zero replicas means any inbound request receives an immediate error rather than waiting for cold provisioning.

To keep production APIs responsive, teams must balance asymmetric timing windows. Keeping scale_up_window tight at 60 seconds prevents missed traffic spikes. Extending scale_down_window to 300 seconds prevents premature scaling down right before another burst. For mission-critical endpoints, running at least one active replica remains mandatory.