Artificial Intelligence

Prefix cache tips won't fix agents alone

Together AI and Ollama say prefix caching decides agent speed, but small prompt changes can wipe out the savings.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Server GPUs handling long context agent inference requests
Credit: Together AI

Together AI teamed with Ollama to explain how to keep long coding-agent sessions fast without reprocessing the full transcript every turn. The obvious reading is that prefix caching solves agentic inference.

I do not buy that it solves it. I think it exposes how brittle agents still are.

The claim itself is accurate as far as it goes. Zain Hasan and Parth Sareen of Ollama describe an agent pattern with large input and small output, where each turn resends the system prompt, tools, files and history to get back a short tool call. Prefill dominates that work, they write, while decode only streams a few dozen tokens. Ten equal turns without caching process 55 units of prompt versus 10 with caching, and 100 turns process 5,050 versus 100, according to their idealized math.

Cache matches from the first token and stops at the first change.

That one rule carries the whole harness section, and it is useful. Put a timestamp, git status or changing directory in the system prompt and every turn becomes a full re-read, the authors warn. Load a new tool definition into the top of the prompt and you invalidate everything after it. Raise reasoning effort mid-run in GLM-5.3-Flash and you do the same, because that model writes effort into the first system block. The fix list is strict: keep the system prompt stable, append tool outputs at the end, truncate files with explicit ranges, and pick effort once. Their timezone example makes the point, when an agent reads lines 1 to 100 of a 640-line calendar.py and misses convert_timezone at line 420.

To be fair, the harness advice is concrete and most teams break at least one of these rules. What is missing is proof on the engine side. The published text stops just as it reaches whether the cache is still resident and routed to the same GPU. There are no cache-hit rates, no time-to-first-token curves, no cost per 100-turn session. Scale makes that gap matter, since community deployments list GLM-5.3 at 744B parameters and GLM-5.3-Flash at 321B, with NVIDIA now listing an FP4 local quant for self-hosting. Caching a 300B-plus session across 100 turns is not free memory.

If you want long agents, fix your harness before you rent bigger GPUs.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsTogether AIOllamaGLM-5.3GLM-5.3-FlashDeepSeek-V4.1-FlashKimi K3NVIDIA

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.