OpenAI Triples Benchmark Scores With Simple Settings

OpenAI's advanced AI models struggled on the ARC-AGI-3 benchmark until researchers tweaked two API settings, according to a new OpenAI News publication. The models, including GPT-5.6 Sol, initially scored poorly, with GPT-5.6 Sol managing only 7.8% and GPT-5.5 scoring a mere 0.4%.

This performance on the 2D puzzle game benchmark was perplexing, given the models' success on complex math problems and other games like Pokémon FireRed. The issue wasn't inherent model weakness, but rather the benchmark's generic harness design.

The ARC-AGI-3 benchmark, designed to test AI reasoning without explicit instruction, uses a simple harness to make model shortcomings apparent. This contrasts with commercial developers who optimize harnesses for specific model features.

Memory is Key

The core problem identified was how the benchmark's harness discarded private reasoning after each action. This forced the AI to restart its problem-solving process from scratch every turn, unable to retain its thought process or past actions due to a rolling truncation window.

StartupHub.ai data indicates OpenAI's own platforms, like Codex, score highly on AI capabilities with a score of 47/100, but benchmarks are a different challenge. When OpenAI applied its own OpenAI Responses API, which retains reasoning, GPT-5.6 Sol's performance improved significantly.

With retained reasoning, the model spent less time analyzing each step and developed more coherent strategies, improving its learning over time.

Compaction Cuts Waste

Further gains came from enabling 'compaction,' another feature of the OpenAI Responses API. This setting replaces the benchmark's rolling truncation, which discards older information as context grows.

Compaction allowed GPT-5.6 Sol to better preserve learned game mechanics over longer sessions. This resulted in higher scores and, notably, a 6x reduction in output tokens compared to the original harness.

The combination of retained reasoning and compaction tripled scores on the public task set, boosting GPT-5.6 Sol's performance from 13.3% to 38.3% against the official harness. This is still below the estimated human average of 48%, but represents a massive leap.

Benchmark Realities

OpenAI emphasizes that evaluations rarely test models in isolation. They are also influenced by API settings and harness design.

For developers aiming for peak performance, OpenAI recommends using their Responses API with retained reasoning and compaction, mirroring their production setups for ChatGPT and Codex.

The findings serve as a reminder that benchmark results must be interpreted with an understanding of the evaluation environment. Anyone looking to test their own AI against these challenges can try the ARC-AGI-3 benchmark at arcprize.org/tasks.