Coding Agent Inference Benchmark Revealed

Traditional inference benchmarks often miss the mark for production AI. Together AI has released a new benchmark designed to stress-test large language models (LLMs) under the demanding conditions of coding agent workloads. This approach prioritizes performance not just at peak, but under sustained, high-traffic scenarios.

The core of the challenge lies in simulating how dozens or hundreds of concurrent requests interact. These requests compete for critical resources like KV cache, memory bandwidth, and GPU cycles. What matters most is how every user experiences performance degradation as the system reaches its limits.

The Coding Agent Workload

Coding agent requests are characterized by large input contexts, often tens of thousands of tokens, representing files, conversation history, and retrieved code snippets. While output lengths are typically bounded, the sheer volume of concurrent requests creates significant pressure.

Together AI's benchmark models this by using prompt lengths ranging from approximately 45,000 to 200,000 tokens, with average generation lengths around 450 tokens. Key metrics include tokens per minute (TPM), tokens per second per user (TPS), and Time to First Token (TTFT).

Why TTFT is King for Developers

For coding agents, TTFT is paramount. The delay between a developer submitting a request and seeing the first token stream directly impacts perceived speed and usability. While output speed is important, a responsive initial stream builds crucial trust.

The benchmark specifically stresses concurrent long-context handling. When numerous developers send requests with extensive context (80k+ tokens), KV cache pressure escalates, leading to increased prefill latency and degraded TTFT.

Optimizing for Prefill-Heavy Outputs

Unlike tasks requiring long, sustained decoding (like document summarization), coding agents often involve short, bursty output generation. This means engines optimized for long decode runs may not perform optimally. The benchmark is designed to highlight these differences.

Methodology and Results

The benchmark was run using 4 NVIDIA B200 GPUs per engine, with SGLang requiring 8 GPUs due to higher memory demands. Together AI's Inference Engine, powered by optimizations like ThunderMLA and custom kernel rewrites, demonstrated superior performance.

At 625 TPM per GPU (2.5 million TPM total), Together AI's engine delivered 31% more TPS than TensorRT-LLM. Crucially, it maintained a TTFT under 1 second, while TensorRT-LLM's TTFT exceeded 1 second. SGLang, running on 8 GPUs, showed a TTFT of 5.1 seconds.

This translates to a system that remains functional at loads where other engines degrade significantly.

Cost and Quality Compared

The benchmark results were based on the Kimi K2.5 model. The newer Kimi K2.6, available on Together AI, rivals or surpasses Claude Opus 4.6 on key coding benchmarks like SWE-Bench Pro and Terminal-Bench.

The cost savings are substantial. For a typical request, Kimi K2.6 on Together AI costs $0.108, compared to $0.451 for Claude Opus 4.6 – a 76% reduction. This could save a 30-person engineering team approximately $440,000 annually on inference costs.

This is version one of the benchmark, with plans for continuous updates to track optimization gains. According to Together AI, the goal is transparency in measuring real-world performance.