Artificial Intelligence

Together's free cache is not really free

Together AI promises 20x warm reads with a free GPU cache, but cold starts, region and NVMe limits decide the real win.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

GPU cluster nodes reading from local NVMe cache
Credit: Together AI

Together AI launched Together Storage Accelerator as a free cache built into Together GPU Clusters. It runs on every GPU node and needs no attacker, no patch and no remote exploit because it is performance infrastructure, not a vulnerability fix.

The obvious reading is free 20x storage speed for AI training. I am not buying free as the full story.

Warm reads hit up to 3.4 GB/s per node on an H100 system, about 20 times faster than cross-cloud pulls, the company said. A 2 GiB object from Cloudflare R2 dropped from 13 to 15 seconds to 0.64 to 0.72 seconds in tests on October 28 in us-central, the company said. Single-stream clients wrote to /dev/null or node NVMe and checksums were MD5-verified, the company said.

That test is narrow by design. Cold reads ran within 5% of a direct download, the company said, so you only win after the first fetch warms cluster NVMe. Small files fell from 320 ms and 680 ms to 5 ms and 8 ms warm, about 60 to 80 times lower latency, the company said.

For the obvious reading to hold, your jobs must reread the same bytes, your NVMe must hold them, and your origin must stay slow. Together AI itself noted the same Hugging Face download ran five times faster from us-central than from ca-west, the company said, which means the 20x claim moves a lot with region, object size and network path. That caveat matters.

To be fair, the egress math is real. AWS egress runs up to $0.09/GB, the company said, and one origin fetch for a 70 GB checkpoint shared across 200 nodes beats 200 fetches every time. Setup is one environment variable for S3 and Hugging Face Hub with no new SDK, the company said.

Context helps too. Rackspace and Together AI announced a multiyear inference infrastructure agreement, with deployment planned for 2027 subject to conditions, per CASRAI reporting. Together Storage Accelerator looks like the unglamorous plumbing for that larger capacity bet: keep GPUs fed before adding more GPUs.

T4, Together AI's managed S3-compatible store, remains in limited preview and requires sales contact for access, the company said, and the ASTRA caching technology behind the feature still awaits its promised engineering deep dive. If you run Together GPU Clusters, turn it on and watch hit rates before you redesign anything. Do not move your source of truth for a cache.

Mentioned in this story

Markets

CloudflareNYSE:NET

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsTogether AITogether Storage AcceleratorASTRAT4H100Cloudflare R2Hugging FaceRackspace

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.