Same B200 can deliver 50% more output depending on operator

Oliver Kennon, research analyst at SemiAnalysis, told AI Engineer that the same Nvidia B200 can deliver about 50% more usable output depending on who runs it. In his measured example both deployments served the same model at roughly 8,000 input tokens and 1,000 output tokens with a P50 target of 100 output tokens per second, yet one serving stack, the software that handles the model responses, hit about 200 tokens per second at P90 while the other hit about 300, according to the InferenceX calculator Kennon cited.

Kennon frames the operator, the team that chooses the serving software, tunes settings and decides how many requests run together, as a multiplier on hardware and electricity. In a commodity market producers compete to deliver comparable work, so a lower cost to deliver the same usable intelligence gives more room when prices fall, and Kennon argued price per token for comparable quality is a better proxy for delivered output than the familiar dollars per GPU hour, which he called input cost, not output price.

The difference hits the bill directly.

At the same hourly rent, Kennon said 1.5 times the output means proportionally lower spend per acceptable token, so an inference producer buying hours or an owner spreading equipment and running costs over output keeps more of the price. Useful intelligence, he said, has to be good enough, fast enough, available when needed and kept within access boundaries, and he pointed to published security work where monitoring access exposed another customer's logs and metrics on a fast cluster, a failure he called horrifically failed and unfortunately common that no chip metric would catch and that sat with the operator to prevent.

To assess who can deliver, Kennon pointed to SemiAnalysis ClusterMAX 3.0, which rates managed GPU clusters on provisioning, reliability, monitoring and security through hands-on testing, documentation and customer feedback, and which in its latest release covered 77 providers according to SemiAnalysis. He stressed the rating applies to managed clusters, not bare metal where the customer does more of the operating work, and warned better operations requires investment in testing, safe deployment and rollback, so one good benchmark does not establish a lasting advantage, you have to earn it every day while competitors keep improving.

That operating gap shapes how a compute market can be graded and hedged. Kennon said a grade, a description of a compute service and what buyers can expect, is needed to know which B200 hour offers are comparable, while a rental price benchmark, a summary of GPU hour prices in a defined market, is not production cost and is shaped by service scope, contract terms and how prices are sampled. A futures contract with cash settlement can help offset rental price moves if the reference tracks your actual rent, Kennon said, but it does not fix a failed deployment or guarantee a customer pays, which is why rating the operator is a building block for a credible market, not a substitute for it.