AI Research

Crusoe let AI tune GLM-5.3 serving

Crusoe let a dstack agent design GLM-5.3 serving on 64 GB200 GPUs, lifting throughput 134% and beating the OpenRouter median.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

NVIDIA GB200 NVL72 rack serving GLM-5.3 coding agents
Credit: Crusoe Blog

Michael YenChi Ho, Crusoe's director of technical product marketing, says his team stopped hand-tuning inference and let a dstack preset agent design the GLM-5.3 deployment on 16 GB200 VMs.

It worked.

On 64 Blackwell GPUs in one NVLink domain, the agent lifted output from 2,188 to 5,114 tokens per second. That is up 134%, according to the Crusoe test. It cut median time to first token from 383 ms to 293 ms. At 48 coding agents, the median first token landed in 0.29 seconds against 0.99 seconds for the median of 47 OpenRouter endpoints, the company said.

Martin Cala, Crusoe staff solutions engineer, ran the trials with Andrey Cheptsov, founder and CEO of dstack, who supplied the preset harness. Cala reported that almost all of the gain came from layout choices, not engine tuning. Those choices were whether to split prefill and decode, how many engines to run, how wide each engine is, and where the KV cache lives.

Left free to choose, the agent picked seven two-node TP8 engines with 140 GB of host memory cache per GPU rank. That detail matters because each node pairs its GPUs to 881 GB of host memory over NVLink-C2C at 900 GB/s, according to the test. A GB200 puts two Blackwell GPUs with one Grace CPU on a superchip, while a B200 ships as eight GPUs in an HGX server. A GB200 or GB300 normally meets as 72 GPUs in an NVL72 rack.

The result follows Crusoe tying Nvidia at 72 GPUs in an earlier cloud check. The cookbook baseline adapted for GB200 used seven TP8 engines behind an SGLang gateway. The pinned-disaggregation baseline doubled a SemiAnalysis 128-agent config to fit 64 chips. Aggregated serving won at 48 agents, the team said, while prefill/decode disaggregation kept scaling to 192.

Cala cautioned that the two scale sweeps used different pacing, so direct comparison needs care. The disaggregation sweep capped idle gaps at 5 seconds and ran 929 seconds per point. The aggregated sweep used SemiAnalysis pacing at 3,625 seconds. Requests in flight per chip were close at 2.07 against 2.27, the company said, but the traffic was not identical.

Who pays for the extra cache movement when 96% of each prompt is a repeat, and does the winning layout hold once that overlap drops?

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsCrusoeGLM-5.3NVIDIA GB200 NVL72dstackSGLangOpenRouterSemiAnalysis

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.