API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces, an audit by Jennifer Wang, Joachim Baumann, Daniel E. Ho and Sanmi Koyejo, finds that API measurements do not faithfully predict chatbot behavior according to arXiv. The study evaluates ChatGPT, Claude and Gemini across seven systems and nine benchmarks spanning general capability, social bias and sycophancy.
On average, API evaluations scored 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test-retest agreement than corresponding interface evaluations.
For ChatGPT the API versus interface gap exceeded the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation.
The authors varied system prompts, sampling parameters and reasoning settings to see if exposed API controls could reproduce interface behavior, and found they shifted behavior in some cases but did not reliably eliminate the gap.
That documents what the authors call a context-validity gap: measurements obtained through APIs do not necessarily generalize to deployed interfaces, which complicates using API evaluations as proxies for deployed systems. The divergence is familiar to practitioners outside the study, for example OpenAI’s GPT-4o supports up to 128K tokens in the latest API though the consumer ChatGPT interface usually has a smaller limit.