Databricks AI Classify Beats LLMs on Cost

Mapping documents to massive taxonomies of over 100,000 labels is a persistent challenge. Traditional methods like regex and basic machine learning struggle with maintenance and context limits. Databricks proposes a novel solution that pairs vector search with its Databricks AI Classify function, offering a more accurate and cost-effective approach.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

Databricks

$190.0B

A unified data analytics and AI platform built on the lakehouse architecture.

This new method tackles complex use cases such as biomedical entity linking, vendor normalization, and company deduplication. These tasks require matching freeform text against extensive sets of predefined labels, a problem that existing tools often fail to address efficiently at scale.

The Limitations of Current Approaches

Existing solutions, including custom regex rules and keyword matching, prove brittle. They break easily with real-world data variations and require constant updates as taxonomies evolve. Training supervised machine learning classifiers is hampered by sparse and skewed ground truth data, making it difficult to accurately classify rare labels.

More recently, large language models (LLMs) have been explored. However, at the scale of hundreds of thousands of labels, LLMs hit context window limitations and can hallucinate non-existent labels. Passing the entire taxonomy to a frontier model is also prohibitively expensive for high-volume workloads.

Databricks' Hybrid Solution

The Databricks approach involves a two-step process. First, vector search is employed to embed both the document and each label, using a Qwen3-Embedding-8B model. This generates semantic and lexical scores to identify a shortlist of candidate labels.

Second, the Databricks AI Classify function is applied, but only to this reduced shortlist. This significantly narrows the scope for the AI model, improving accuracy and dramatically reducing computational cost.

Performance and Cost Savings

Benchmarking across three datasets, Transactions, Companies, and MedMentions, revealed compelling results. The AI Classify workflow consistently outperformed direct LLM calls. It achieved an average accuracy of 0.81, compared to 0.76 for the next-best model, Gemini 3.5 Flash, all at approximately one-hundredth of the per-document cost.

Vector search alone, while incredibly cost-efficient, lagged significantly in accuracy, scoring over twenty points below the hybrid approach. The direct LLM approach struggled with larger taxonomies, often unable to fit the full label set within context windows.

A Scalable Future for Classification

This combined vector search and AI Classify workflow offers a robust solution for scaling document classification to handle massive label sets. It provides a balance of accuracy, cost-effectiveness, and maintainability, crucial for production environments.

When taxonomies change, retired labels can be removed and new ones embedded seamlessly. This eliminates the need for model retraining or redeployment, streamlining the update process.

For organizations grappling with large-scale classification tasks, implementing this workflow is a pragmatic starting point, as detailed in a Databricks blog post.