NVIDIA says memory hierarchy, not brute HBM capacity, will decide who wins agentic inference, and I think the NVIDIA Vera Rubin NVL72 platform proves it.
The pitch is efficiency over expansion, and for once the numbers back up the marketing.
NVIDIA says HBM capacity per GPU rose roughly 3% from Blackwell Ultra to Vera Rubin while agentic throughput per dollar rose up to 30x on SemiAnalysis AgentX at 125 tokens per second per user.
That last figure matters because NVIDIA frames the AI factory as a hierarchy where HBM is the fastest, most expensive, most contested tier for weights and KV cache.
Below HBM sit CPU memory, local SSDs and shared storage, each trading speed for capacity at lower cost per bit, according to NVIDIA.
NVIDIA says frontier models now carry trillions of parameters and agentic trajectories can process up to 3 million tokens, with RAG adding even more retained context.
HBM becomes the bottleneck because it must hold both model weights and KV cache while bandwidth caps how fast data moves, a limit NVIDIA calls the memory wall.
NVIDIA compares HBM to a warehouse where bandwidth is the loading dock, and a bigger warehouse without more docks just raises cost and power without speeding delivery.
The answer in NVIDIA's stack is full-stack codesign that shrinks what must be stored, places it on the right tier, and prefetches it before the GPU needs it.
For weights, NVIDIA points to NVFP4, which cuts footprint versus FP8 on Hopper, and to distribution across up to 72 GPUs in an NVL72 domain so each GPU holds less.
Recent models including FLUX.2 klein from Black Forest Labs, Inkling, Beam 1.0, Meta Muse Spark, Step 5 Preview and Nemotron 3.5 Super VL have shipped NVFP4 checkpoints, according to NVIDIA.
For KV cache, NVIDIA says NVFP4 also shrinks cache footprint and Vera CPU helps by executing tools faster and parking paused context in large LPDDR over NVLink-C2C.
When context sits idle, Dynamo inference software moves it to CMX context memory storage, a shared tier built for capacity at far lower cost per bit than HBM, according to NVIDIA.
CMX is shared across servers and skips long-term data protection overhead, unlike local SSDs or enterprise storage, and partners include Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA, according to NVIDIA.
Dynamo's KV Cache Reuse then manages cache as one system resource across GPU memory, CPU memory and storage so computed context can be reused, according to NVIDIA.
For retrieval, Nemotron Retriever produces smaller embeddings with higher accuracy while cuVS builds the index across CPU and storage so the knowledge base never occupies HBM, according to NVIDIA.
The net claim is fewer bytes in HBM for weights and idle context, and more concurrent users served, with GB300 NVL72 delivering up to 10x throughput per dollar versus H200 at 100 tokens per second per user on the same AgentX test while roughly doubling HBM per GPU, according to NVIDIA.
The Vera Rubin NVL72 itself is a rack-scale system with 72 Rubin GPUs and 36 Vera CPUs in a single liquid-cooled rack connected by NVLink 6, per Thundercompute's November overview.
That density is why StartupHub.ai data shows the competitive pressure in rack-scale AI hardware is intensifying around co-designed CPU-GPU-memory systems rather than standalone accelerators.
That pressure directly touches rivals StartupHub.ai tracks including AMD and Intel on the silicon side and system builders like Dell Technologies and HPE who are also listed as CMX partners.
The concession is fair: NVIDIA still leans on partners for memory supply and admits supply tightness and rising prices will persist near term, so hierarchy efficiency cannot fully hide cost if HBM pricing spikes further.
If you are building inference infra today, bet on software that squeezes more tokens per watt from the memory you already have, not on waiting for a bigger HBM allocation.
