Databricks Boosts AI Agents with Chart Data

Databricks researchers have developed a novel method to enhance AI agent comprehension of complex documents, particularly those containing charts and figures. This advancement focuses on structured chart extraction, a technique that transforms visual data into machine-readable formats, thereby improving the accuracy of information retrieval and question answering for AI agents. The findings, detailed in a recent Databricks AI Research blog post, suggest a significant leap forward in making AI more effective in enterprise settings where visual data is prevalent.

Companies working on this

StartupHub profiles of the companies this article names, with funding and a one-liner from our database.

Databricks

$190.0B

A unified data analytics and AI platform built on the lakehouse architecture.

The core challenge addressed is the struggle AI agents face when interpreting charts and graphs, a common hurdle that limits their utility in analyzing proprietary business documents. Traditional text-based retrieval systems often miss critical numerical data embedded within these visuals. Databricks’s solution involves using its proprietary ai_parse_document function to extract chart data as structured JSON. This extracted data is then integrated into the retrieval index, making it directly accessible to agents.

Improving Retrieval Accuracy

A key experiment highlighted in the research involved asking AI agents to identify local maxima on a chart. A frontier agent, given only the image, failed to provide the correct answer. In contrast, Databricks Genie, utilizing the structured chart extraction, correctly identified the value. This demonstrates the immediate impact of structured data on agent performance.

The research evaluated this approach on two datasets: a subset of the ViDoRe V3 benchmark, known for its visually rich documents, and a custom synthetic dataset named Chart-RAG. The results indicated that enriching retrieval indexes with structured chart JSON significantly improved both answer quality and retrieval metrics like Hit Rate@10 and nDCG@10 across both datasets.

JSON vs. Multimodal Embeddings

Databricks compared its structured chart extraction method, powered by a lightweight 300 million parameter text embedding model, against several large multimodal embedding models. The structured JSON approach, even when augmented with the top three retrieved images, achieved competitive answer accuracy. Specifically, it reached 75.9% correctness on ViDoRe V3 and 75.1% on Chart-RAG. This performance rivals, and in some cases exceeds, that of larger multimodal models.

This efficiency is a critical advantage. The Databricks approach offers a roughly 10x smaller and simpler alternative to complex multi-vector multimodal architectures. It promises lower indexing and retrieval overhead, making it a more scalable solution for enterprises.

Broader Industry Impact

This development is particularly relevant for industries like financial services, where charts and graphs are integral to reports and analyses. By enabling AI agents to accurately interpret this data, companies can unlock deeper insights and automate more complex analytical tasks. The research also points to the growing importance of integrating structured data extraction within retrieval-augmented generation (RAG) pipelines, a foundational technique for many modern AI applications.

Databricks, a company with a StartupHub score of 82/100 and VERIFIED financials including $5B raised in strategic financing in 2026, valuing it at $190B post-money, continues to push the boundaries of data and AI integration. Its approach contrasts with competitors like Snowflake (score 73/100), which also focus on data warehousing and analytics but may not have the same depth in proprietary AI research for document understanding.

The future implications extend to the development of more capable AI assistants that can truly act as co-pilots for business users. This advancement in structured chart extraction addresses a critical gap in current AI capabilities, paving the way for more reliable and insightful AI-driven analysis of enterprise information. The structured chart JSON enrichment for ai_parse_document is expected to be available soon, integrating directly into Databricks's Genie platform.