The 2 Gigawatt AI Inference Problem

It’s come time to read the meter.

Every answer from an AI system draws power, time, and money. When a feature goes viral, those draws add up like clock ticks on a utility meter. Inference, the act of running a trained model to produce tokens, images, audio, or video, is no longer a rounding error. It is the day‑to‑day business of AI: what users feel as speed, what operators experience as throughput and tail latency, and what finance sees as a recurring bill.

OpenAI offers a clear sense of scale. ChatGPT usage sits in the hundreds of millions of users per month, with figures of 700, 800 million cited publicly. Its APIs have been described as processing about 8 billion tokens per minute across endpoints. Each token is a small unit of work: memory moves, matrix multiplies, cache reads, and network hops. At this altitude, shaving 20, 50 milliseconds from the decode loop or reducing compute per token by 10, 20 percent can be the difference between a product that feels instant and one that makes people wait.

What inference is, and why it sped up

First, a quick grounding: Training is about learning; inference is about doing.

Inference happens right now, for this user, for this request. Three constraints govern the experience and the economics: latency (how long someone waits), throughput and its tail behavior (how many people you can serve at once and how predictable the slowest turns are), and unit cost (dollars per request, often summarized as dollars per million tokens). The serving playbook pushes on those constraints in four practical ways: reuse work already done, avoid work that does not need to be done, place the work closer to the user, and keep accelerators busy without letting the slowest requests stretch the tail.

In plain terms, that playbook starts with model compression: use quantization (INT8/INT4, FP8/NF4) so weights and activations take fewer bits, fit in memory, and run faster. Second, adaptive compute: route easy questions to small models and escalate only when needed; inside a large model, use mixture‑of‑experts so only a subset of “experts” activates per token rather than the entire network. Third, decoding and attention efficiency: use speculative decoding so a small “drafter” proposes a short run of tokens and the target model verifies them in one pass; maintain a KV cache so the model does not recompute the entire history at every step; adopt attention kernels that move fewer bytes. Fourth, system and hardware optimization: employ iteration or continuous batching so new requests can join in‑flight batches between decode steps; use better kernels to reduce memory traffic; and place latency‑sensitive models near users.

For years, most concern focused on training. That concern was justified; the numbers are concrete and visible. Epoch AI has tallied training compute growth at roughly 4× per year on average since 2010. By mid‑2025, more than 30 frontier models had passed the 1e25 FLOP threshold. Run durations have stretched from months toward a year at the very frontier. These are counts, not impressions, and they matter for serving. Longer context windows, richer modalities, and higher ceilings on reasoning make products more capable, but they also raise the steady‑state work a service must perform. Training happens in bursts; inference runs all day. In other words, once a model ships, the recurring bill starts rather than stops.

How people use these systems is shifting as well.

Agentic AI turns a single request into many small steps. Ask for a working prototype, a code refactor, a research brief, or a product plan, and the system does not make one pass. It plans sub‑tasks, retrieves information, calls tools, writes and tests code, packages artifacts, performs checks, deploys, and revises. Each of those steps can involve multiple model calls. A single prompt can quietly trigger thousands of inferences, some in sequence and some in parallel. And this recursiveness is not limited to coding. Everyday work, summarizing meetings, preparing proposals, coordinating schedules, producing first drafts, naturally expands into chains of retrieval, drafting, review, and revision. That compounds the depth and activity we should expect as agentic coworkers become normal in knowledge work. For readers who want a snapshot of what people actually use, the a16z lists of fast‑rising AI apps and their analysis on agentic coworkers show how quickly these patterns are moving from demos into daily usage.

To be fair, per-call inference costs are plummeting, dropping up to 1,000x since 2022 thanks to efficient models and new GPUS, like Blackwell, making advanced AI more accessible. Yet, this is offset by surging call intensity in Agentic AI systems, spawning recursive chains of thousands of back-to-back inferences.

Now the curve steepens further with multimodal.

What began with text now routinely includes images and audio, and increasingly, full video, which turns a steady stream of tokens into something closer to highway traffic. An image is a grid of pixels; audio is a continuous waveform; video multiplies both frame by frame, second by second. Moving from text to video is a step change, not a gentle incline. Serving video well means tiled or paged attention, chunked diffusion, frame‑level caching, and distributed schedulers that keep accelerators saturated while holding tails in check. The same serving principles still apply, reuse, avoid, place, batch, but the volume of computation per second is simply larger.

A practical response at the application layer is to arrange chains so the system avoids paying “big” at every link. Emergent illustrates this approach. Emergent is an Agentic coding platform, allowing consumers to build production grade apps via natural-language and backend AI agents.