#SWE-bench

3 articles with this tag

Smarter models didn't kill the harness
Technology

Smarter models didn't kill the harness

Jedidiah Katz says frontier models absorbed coding tricks, but the agent harness still controls trust, cloud runtime and cost.

about 2 hours ago
AI Coding Benchmark Scores Skewed by Infrastructure
Artificial Intelligence

AI Coding Benchmark Scores Skewed by Infrastructure

Infrastructure configuration, not just AI model prowess, can significantly skew benchmark results, complicating deployment decisions.

7 months ago
AI Coding Tests Flawed by Infrastructure Noise
Artificial Intelligence

AI Coding Tests Flawed by Infrastructure Noise

The infrastructure powering AI coding tests can significantly inflate or deflate model scores, potentially masking true capabilities and misleading deployment decisions.

9 months ago