AI Research

Databricks says context beats bigger models

Databricks says Genie with Ontology hit 91% accuracy with less than half the tokens by reading notebooks, not just tables.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

2 min read

Databricks Genie benchmark comparing accuracy and tokens
Credit: Databricks

Business questions fail when the definition lives outside the table. Databricks says its answer, built with Turing, hit a 91% pass rate on live workspace tasks, against 45% and 58% for two frontier agents.

Tables alone do not know the rule.

Genie Ontology is the enterprise context layer that extracts business definitions from work teams already did. It scans notebooks, dashboards, saved SQL queries and documents, then surfaces the relevant snippet when Genie answers. That retrieval is why Databricks credits the system with 431,829 tokens per run and 100.3 seconds wall time, according to the company, against 889,019 and 434.6 seconds for Agent A and 1,051,070 and 306.5 seconds for Agent B. Each task ran five times in the same live Databricks workspace.

The test company is Calderwick Energy, a fictional UK supplier with hundreds of tables and the usual mess. The largest table holds around 120 million rows of half-hourly meter data, plus copied notebooks, frozen reporting tables and old docs, according to Databricks. One task asks for the Commercial planning rate for FY24/25. The bill cost stack table shows 0.31 pence per kWh, which reflects FY25/26 assumptions. The notebook that builds the mart, 45_mart_bill_cost_stack, records the history as 0.52 for FY24/25, 0.31 for FY25/26 and 0.45 draft for FY26/27. Genie retrieved that snippet and returned 0.52 in all five runs, while the other agents returned 0.31 or said no year-specific rate was documented, according to Databricks.

The frontier agents ran through a Databricks toolbelt, not natively. The toolbelt excluded Genie MCP, the Model Context Protocol interface that lets outside agents call Genie tools, to keep the comparison on retrieval and reasoning. Unity Catalog is the Databricks governance layer that controls who can see what data, and all agents worked under it. The same setup explains the limit Databricks states directly. In new deployments with little workspace knowledge, or where tables and schemas are enough, the gap between agents narrows.

That context layer is now showing up outside Databricks itself. The company launched native Genie plugins for ChatGPT, Claude, Microsoft Copilot Cowork and Grok Bot, giving those assistants governed access to the same business context, as reported by Unite.AI. Watch whether teams use Genie in the UI, through Genie MCP or inside those hosts, and whether their own notebooks hold the definitions that decide the answer.

Mentioned in this story

Markets

MicrosoftNASDAQ:MSFT

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsDatabricksGenieGenie OntologyTuringCalderwick EnergyChatGPTClaude

StartupHub.ai Staff

Written by

StartupHub.ai StaffEditorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Reviewed by

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.