Blackwell B200 nodes shipped with tens of terabytes of local SSD for spillover, and VAST Data says that model breaks at $38 per GPU-hour scale. The company published the argument with DeepInfer revenue math as proof.
KV cache lives in three tiers, per VAST Data. GPU HBM at 80 to 288 GB per GPU runs tokens, host DRAM at 512 GB to 4 TB per node holds overflow, and NVMe SSDs hold warm history. For GLM-5.3-Flash at about 16 KiB per token, an 8x B200 node holds about 50.0 million tokens in 800 GB of HBM and about 62.5 million tokens in 1 TB of DRAM, then jumps to about 1.95 billion tokens with 30 TB of flash.
Local flash strands those hits.
A prefix on local SSD only helps if the next turn lands on that same node, per VAST Data, so operators use sticky routing that hurts load balancing. NVIDIA Dynamo documents that tradeoff as faster time to first token against worse inter-token latency, per VAST Data, and only a matching prefix can be reused while a cache built by one model is useless to another, per Respan. NVIDIA, DeepSeek and Alibaba Cloud have each moved toward shared caches that any prefill engine can hit.
Wear is the harder limit, per VAST Data. Light-use drives at 0.8 to 1.5 drive writes per day can die in months under KV churn, and University of Chicago, Samsung and TensorMesh research puts even a 30 TB enterprise SSD under one year, per VAST Data. One Kimi-K3 trace wrote about 42 TB to a 1 TB SSD in two hours for a projected 3.5 days of life against a five-year rating, per VAST Data, while GPU nodes last five to six years and a RAID0 stripe fails with one drive.
Agentic sessions make the gap wider, per VAST Data. A 1M-token GLM-5.3 session needs about 16 GB of KV, so 800 GB of HBM holds only about 50 concurrent sessions, while DeepSeek V4.1 Flash compresses to about 890 bytes per token and DeepInfer retains about 14.4 TB logical per GPU for days, per VAST Data. Shared NVMe-over-Fabrics writes once for cluster-wide use, levels wear across a pool, and leaves local NVMe for weights and container state, per VAST Data. The scheduled test is 800 Gbps East-West fabrics replacing 400 Gbps in NVIDIA Cloud Partner designs.
