Databricks Puts AI Search Inside Postgres

Databricks is injecting advanced search capabilities directly into its Lakebase Postgres database with the unveiling of Lakebase Search. This new feature, now in beta on AWS and Azure, aims to streamline the development of AI agents by building native retrieval functions into the data backend.

The system utilizes two new Postgres extensions, lakebase_vector and lakebase_text, to provide hybrid vector and full-text search. This integration allows the entire AI agent loop, from retrieval and reasoning to action and memory, to operate on a single data foundation.

Agents Demand a New Kind of Search

Unlike traditional search engines that query static data, AI agents treat search as a live operational workload. They continuously write new information to memory and require instant access to it in subsequent turns.

This dynamic creates a read/write loop where freshly generated insights must be immediately searchable. Existing solutions often struggle with the scale and cost demands of this continuous indexing and retrieval.

The Cost of Vector Bloat

Search workloads, particularly vector search, are known for creating significant data bloat. A small text file can expand considerably when converted into high-dimensional embeddings, leading to massive storage and memory requirements.

Traditional memory-bound index architectures, like HNSW, become prohibitively expensive when hosting large, multi-tenant datasets that are mostly inactive.

Lakebase Search: Tiered Storage for Efficiency

Databricks' Lakebase architecture, which places data in cheap object storage with a tiered cache, provides a foundation for cost-effective search. Lakebase Search builds upon this by introducing a purpose-built index designed for tiered storage.

This approach allows the active working set of data to reside in fast local caches (RAM and NVMe), while the bulk of cold data remains in inexpensive object storage. This significantly reduces costs by only requiring expensive memory for frequently accessed data.

Per terabyte per month, this tiered approach offers substantial savings compared to keeping entire indexes in RAM.

Native Postgres Extensions for Scale

Lakebase Search introduces lakebase_vector for compressed vector indexes and lakebase_text for efficient full-text search. lakebase_vector uses techniques like Randomized Binary Quantization to achieve up to 32x compression, enabling indexes to scale to over a billion vectors while fitting within manageable memory footprints.

lakebase_text offers true BM25 relevance ranking without the memory bloat associated with traditional GIN indexes in Postgres. This allows for hybrid search queries that combine vector similarity and keyword relevance directly within a single SQL statement.

Performance Benchmarks

Benchmarks on the LAION-100M dataset show Lakebase Search delivering high recall (0.955) with low latency (30 ms) on a significantly smaller instance compared to traditional memory-bound solutions.

A 100 million vector index requiring 512 GB of RAM for a standard pgvector HNSW setup runs on a 192 GB instance with Lakebase, with cold-cache query times dropping from minutes to just over a second.

Agent-First Ergonomics

By integrating search directly into Lakebase Postgres, Databricks simplifies agent development. Developers can consolidate memory and context retrieval into a single backend, leveraging existing Postgres tools and connectors.

The ability to perform hybrid searches, join with operational tables, and filter results within a single SQL query streamlines application logic.

Furthermore, Lakebase Search facilitates continuous experimentation by allowing cheap branching of datasets and out-of-band index building. It also enables the creation of thousands of isolated, dedicated search corpora for individual agents, shifting search from a static snapshot to a dynamic, transactional workflow.

Lakebase Search is available now in beta on AWS and Azure.