Together AI says 80% of traffic on OpenRouter goes to models released in the last three months, a rotation speed it cites to justify Polymorphic Inference. Authors Simran Arora, SVP of Research, AI Systems, and Dan Fu, Founding SVP of Research, introduced the approach on Nov. 4 alongside Together Compiler, an agent swarm paired with a persistent engine that the company says continuously recompiles production endpoints as models, traffic shapes and hardware change. FinancialContent, covering the launch, described Together Compiler as the first step toward a self-improving inference stack, a form of recursive self-improvement for inference workflows.
Update, Nov 4, 15:06 UTC · via Together AI
Together AI reported that in less than 24 hours its system achieves "up to 2.6x higher TPS across frontier open source models (Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash)" and "outperforms available open-source megakernels by 1.7x on their workhorse Llama 1B batch size 1 setting." The company said it "sets the state-of-the-art on SemiAnalysis’ InferenceX benchmark for low-latency with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash" enabled by "252x higher experiment throughput compared to baseline agentic performance optimization systems."
Together AI also detailed traffic dynamics, reporting prompts during one country's working hours were "1.4x longer prompts than another country (97k vs 70k)" and that users show an "80% increase in prompt length just two weeks after the release" as they shift to agentic workloads. The post listed authors as Reyna Abhyankar, Dylan Lim, and Simran Arora.
Static configs cannot track that churn.
The company says it now serves trillions of tokens per day and that in October it served more tokens than closed-source labs on OpenRouter for the first time, according to the post. That scale claim matters because the usual open stack, vLLM or SGLang launched with a fixed tuned config, assumes a stable shape. Together AI argues a config that is optimal for one model or batch size is often miscalibrated for another, with MoE kernels as a classic mismatch between random token tests and live traffic.
A third figure underlines diversity. Together AI says more than 40 models are chosen for production, and all of the top ten most popular open models in use today use different architectures, according to the post. The comparison is to the prior era it calls Static Inference, where a single Transformer on a single hardware platform could be hand tuned once with a megakernel like FlashAttention and stay near optimal.
What the numbers do not show is the magnitude of the win. The post says Together Compiler has found further gains on top of already highly optimized code on serverless endpoints, but it discloses no latency, throughput or cost per token deltas, no before and after on B200 versus GB300, and no Vera Rubin results. Without those deltas, it is hard to judge how much Continuous Compilation closes the gap to ideal hand tuned baselines.
The daily brief
Get the free daily brief with the day's AI news, new AI startups and funding rounds every morning.
Join 64,000 other subscribers.
A disclosed gain curve by workload, model family and hardware would change the read, because it would show whether the agent search pays off quickly or only after long tail tuning. Until then the open question is how fast the swarm can match a hand tuned kernel when traffic flips, a new reasoning heavy post-train lands, or a cluster moves to a new accelerator.
