# Together AI says inference must now tune itself _Together AI says 80% of OpenRouter traffic is to models under three months old and launched Together Compiler to continuously retune inference for shifting models and hardware._ **Published:** 2026-11-04 **Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/together-ai-says-inference-must-now-tune-itself --- [Together AI](https://www.together.ai/blog/polymorphic-inference) says 80% of traffic on [OpenRouter](https://www.startuphub.ai/ai-news/startup-news/2026/stripe-buys-openrouter-for-7b-groq-raises-350m) goes to models released in the last three months, a rotation speed it cites to justify Polymorphic Inference. Authors Simran Arora, SVP of Research, AI Systems, and Dan Fu, Founding SVP of Research, introduced the approach on Nov. 4 alongside Together Compiler, an agent swarm paired with a persistent engine that the company says continuously recompiles production endpoints as models, traffic shapes and hardware change. FinancialContent, covering the launch, described Together Compiler as the first step toward a self-improving inference stack, a form of recursive self-improvement for inference workflows. Static configs cannot track that churn. The company says it now serves trillions of tokens per day and that in October it served more tokens than closed-source labs on [OpenRouter](https://www.startuphub.ai/ai-news/startup-news/2026/stripe-buys-openrouter-for-7b-groq-raises-350m) for the first time, according to the post. That scale claim matters because the usual open stack, vLLM or SGLang launched with a fixed tuned config, assumes a stable shape. Together AI argues a config that is optimal for one model or batch size is often miscalibrated for another, with MoE kernels as a classic mismatch between random token tests and live traffic. A third figure underlines diversity. Together AI says more than 40 models are chosen for production, and all of the top ten most popular open models in use today use different architectures, according to the post. The comparison is to the prior era it calls Static Inference, where a single Transformer on a single hardware platform could be hand tuned once with a megakernel like FlashAttention and stay near optimal. What the numbers do not show is the magnitude of the win. The post says Together Compiler has found further gains on top of already highly optimized code on serverless endpoints, but it discloses no latency, throughput or cost per token deltas, no before and after on B200 versus GB300, and no Vera Rubin results. Without those deltas, it is hard to judge how much Continuous Compilation closes the gap to ideal hand tuned baselines. A disclosed gain curve by workload, model family and hardware would change the read, because it would show whether the agent search pays off quickly or only after long tail tuning. Until then the open question is how fast the swarm can match a hand tuned kernel when traffic flips, a new reasoning heavy post-train lands, or a cluster moves to a new accelerator. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory. © StartupHub.ai. All rights reserved. You may not republish this article in full without a license. Search engines and AI research tools may crawl and summarize for reference. Bulk reproduction or model training on this content requires a license. See https://www.startuphub.ai/terms.