Databricks Benchmarks AI Coding Tools

Databricks has developed a rigorous internal benchmark to assess the efficacy of AI coding agents, a crucial step as the company increasingly integrates AI into its software development lifecycle. The initiative aims to understand which tools perform best on real-world coding tasks and how their costs stack up.

The benchmark leverages Databricks’ own extensive codebase, spanning millions of lines across languages like Python, Go, and TypeScript. This approach, detailed on the Databricks blog, ensures the tasks and solutions are directly relevant to the company's engineering challenges.

AI Coding Agent Performance Insights

Key findings reveal that the Pareto frontier, the optimal balance of quality and cost, includes models from OpenAI, Anthropic, and notably, open-source options. GLM 5.2, an open model, demonstrated performance on par with top-tier proprietary models like Opus 4.8 but at a significantly lower per-task cost.

The analysis debunked the assumption that token price directly correlates with overall task cost. Larger, more efficient models often incur lower end-to-end expenses despite higher per-token rates.

Furthermore, the choice of 'harness', the framework used to interact with the model, profoundly impacts both cost and quality. Simple harnesses like Pi sometimes outperformed more complex ones by managing context more effectively.

Building a Realistic Benchmark

Databricks opted for a proprietary benchmark over public ones like SWE-Bench due to concerns about training data contamination and the need for relevance to their specific, multi-language codebase. This ensures the benchmark's results provide actionable insights for their engineering teams.

The benchmark construction involved carefully filtering thousands of recent, human-written pull requests with high-quality test suites. Tasks were then distilled to their core intent, with test suites separated to enable objective evaluation of agent-generated code.

To prevent agents from simply retrieving solutions from Git history, the benchmark implemented sealed Git repositories for each task run. This guards against agents exploiting the codebase's history, ensuring a true test of their coding capabilities.

Future Directions

Databricks plans to continuously expand its benchmark with more tasks, especially complex ones, and automate the selection and tracking of AI coding agent efficiency. This data-driven approach aims to provide engineers with the most effective tools while maintaining flexibility and avoiding vendor lock-in.