# OpenAI Triples Benchmark Scores With Simple Settings _OpenAI found simple API setting tweaks tripled scores on the ARC-AGI-3 benchmark, proving harness design significantly impacts AI evaluation._ **Published:** 2026-07-29 **Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/openai-triples-benchmark-scores-with-simple-settings --- OpenAI's advanced AI models struggled on the ARC-AGI-3 benchmark until researchers tweaked two API settings, according to a new [OpenAI News](https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores) publication. The models, including GPT-5.6 Sol, initially scored poorly, with GPT-5.6 Sol managing only 7.8% and GPT-5.5 scoring a mere 0.4%. OpenAI models struggleDriver GPT-5.6 Sol scored 7.8%, GPT-5.5 only 0.4% on ARC-AGI-3 benchmarkFrom the article 2 mentionsOpenAI's advanced AI models struggled on the ARC-AGI-3 benchmark until researchers tweaked two API settings, according to a new OpenAI News publication.due toGeneric harness designDriverbenchmark's simple harness discarded private reasoning after each actionFrom the article 2 mentionsThe issue wasn't inherent model weakness, but rather the benchmark's generic harness design.Memory is KeyContextAI forced to restart problem-solving from scratch every turnCompaction Cuts WasteContextoptimized memory usage by compacting redundant informationaddressed byTweaked API settingsCoreresearchers adjusted two simple API settings for model interactionFrom the article 2 mentionsOpenAI's advanced AI models struggled on the ARC-AGI-3 benchmark until researchers tweaked two API settings, according to a new OpenAI News publication.led toTripled benchmark scoresOutcomemodels achieved significantly higher scores on the ARC-AGI-3 benchmarkFrom the article 2 mentionsThe combination of retained reasoning and compaction tripled scores on the public task set, boosting GPT-5.6 Sol's performance from 13.3% to 38.3% against the official harness.demonstratesHarness design impactsEffectproving how benchmark harness design significantly affects AI evaluationFrom the article 2 mentionsThey are also influenced by API settings and harness design.illustratesBenchmark RealitiesContexthighlights the difference between academic and commercial benchmark optimizationFrom the article 9 mentionsThis performance on the 2D puzzle game benchmark was perplexing, given the models' success on complex math problems and other games like Pokémon FireRed. This performance on the 2D puzzle game benchmark was perplexing, given the models' success on complex math problems and other games like Pokémon FireRed. The issue wasn't inherent model weakness, but rather the benchmark's generic harness design. The ARC-AGI-3 benchmark, designed to test AI reasoning without explicit instruction, uses a simple harness to make model shortcomings apparent. This contrasts with commercial developers who optimize harnesses for specific model features. ## Memory is Key The core problem identified was how the benchmark's harness discarded private reasoning after each action. This forced the AI to restart its problem-solving process from scratch every turn, unable to retain its thought process or past actions due to a rolling truncation window. StartupHub.ai data indicates OpenAI's own platforms, like Codex, score highly on AI capabilities with a score of 47/100, but benchmarks are a different challenge. When OpenAI applied its own [OpenAI Responses API](/ai-news/ai-research/2026/openai-s-vinoth-govindarajan-on-agent-failures-and-harness-design), which retains reasoning, GPT-5.6 Sol's performance improved significantly. With retained reasoning, the model spent less time analyzing each step and developed more coherent strategies, improving its learning over time. ## Compaction Cuts Waste Further gains came from enabling 'compaction,' another feature of the [OpenAI Responses API](/ai-news/artificial-intelligence/2026/openai-explains-value-maximization-with-gpt-5-6). This setting replaces the benchmark's rolling truncation, which discards older information as context grows. Compaction allowed GPT-5.6 Sol to better preserve learned game mechanics over longer sessions. This resulted in higher scores and, notably, a 6x reduction in output tokens compared to the original harness. The combination of retained reasoning and compaction tripled scores on the public task set, boosting GPT-5.6 Sol's performance from 13.3% to 38.3% against the official harness. This is still below the estimated human average of 48%, but represents a massive leap. ## Benchmark Realities OpenAI emphasizes that evaluations rarely test models in isolation. They are also influenced by API settings and harness design. For developers aiming for peak performance, OpenAI recommends using their Responses API with retained reasoning and compaction, mirroring their production setups for ChatGPT and Codex. The findings serve as a reminder that benchmark results must be interpreted with an understanding of the evaluation environment. Anyone looking to test their own AI against these challenges can try the [ARC-AGI-3 benchmark](/ai-news/ai-research/2026/fran-ois-chollet-on-arc-agi-3-the-future-of-ai-reasoning) at arcprize.org/tasks. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.