Databricks redefines databases

Databricks is pushing a radical rethink of database architecture, moving from the monolithic structures that have long defined systems like Postgres and MySQL to a new paradigm called LTAP. This evolution, detailed on the Databricks blog, aims to solve fundamental issues of data loss, scaling, and performance interference.

Traditional databases store their write-ahead logs (WAL) and data files on a single machine's disk. This monolithic design, while functional, creates inherent risks: data loss due to misconfiguration or hardware failure, expensive read replicas for scaling, and analytical queries that directly degrade transactional performance.

The Monolith's Limits

The core problem lies in the co-location of WAL and data files. Committing a transaction involves writing to the WAL first for speed and safety, then asynchronously updating data files. This makes durability dependent on local disk reliability.

Scaling reads necessitates creating full physical copies (read replicas), a costly and time-consuming process. High availability similarly requires keeping at least one identical standby, doubling infrastructure costs and complicating failover.

Crucially, heavy analytical queries compete for resources with latency-sensitive transactional workloads on the same hardware. This contention can cripple application performance.

Lakebase: Decoupling Compute and Storage

Lakebase fundamentally alters this by externalizing the WAL and data files into independent, cloud-native services. The WAL is managed by a service called SafeKeeper, which uses distributed network replication for durability, eliminating single points of failure.

Data files are handled by PageServer. It streams WAL updates from SafeKeeper and asynchronously materializes changes into low-cost cloud object storage. This effectively turns compute instances into stateless components.

This decoupling unlocks significant benefits: virtually unlimited storage capacity, elastic and serverless compute that can scale to zero, and truly durable writes without the risk of silent data loss.

LTAP: Unifying Transactions and Analytics

The LTAP database architecture takes this a step further. It enables both transactional (OLTP) and analytical (OLAP) workloads to operate on a single, up-to-date copy of the data.

Operational data is stored once in open columnar formats accessible by both Postgres and Lakehouse engines.

This means analytics queries run on the exact same fresh data that transactions have just written, eliminating the need for complex Change Data Capture (CDC) pipelines or maintaining duplicate data copies.

Unlike Hybrid Transactional/Analytical Processing (HTAP) solutions that try to force both workloads into a single engine, LTAP unifies at the storage layer while allowing specialized engines to handle each job optimally.

This innovative approach promises real-time analytics without impacting transactional throughput or incurring the overhead of data synchronization.