Liquid Clustering Replaces Partitioning

For over 15 years, data partitioning has been the default method for organizing data in systems like Hadoop and Hive. However, the demands of modern Lakehouses, serving real-time pipelines and AI agents, outpace the static nature of partitioning. Databricks introduces Liquid Clustering as the successor, a data layout designed for open table formats that sidesteps partitioning's limitations and delivers dramatic improvements.

Partitioning forces users to commit to a physical data organization at table creation, often leading to billions of tiny files or slower query performance. This can result in over-partitioning and small-file problems in over 75% of cases analyzed by Databricks. Liquid Clustering, on the other hand, treats clustering keys as guidance for optimal file organization, allowing keys to be changed or intelligently selected via Automatic Liquid Clustering without costly rewrites. This flexibility addresses issues like data skew, small files, and enables multi-dimensional clustering, all while reducing write amplification.

Debunking Data Layout Myths

Several persistent myths about data layout are holding back adoption of more efficient methods. Databricks aims to debunk these, highlighting why Liquid Clustering is the future.

Myth #1: Partitioning is faster due to directory pruning.

This is false. Modern open table formats like Delta and Iceberg prune at the file level using per-column statistics stored in transaction logs, not by listing directories. Liquid Clustering uses the same mechanism, achieving file-level pruning regardless of directory structure.

Myth #2: Partitioning is better for low-cardinality columns.

Liquid Clustering automatically optimizes for low-cardinality columns, aiming for each file to contain data from a single value. Higher-cardinality columns then provide finer-grained sorting. Benchmarks show a 35% reduction in clustering time and 22% faster queries with this optimization.

Myth #3: Liquid Clustering doesn’t support metadata-only operations.

Liquid Clustering supports metadata-only operations like DELETEs, COUNT, DISTINCT, and GROUP BY queries using per-file statistics. Metadata-only DELETEs on Liquid Clustered tables are approximately 90% faster than full rewrites.

Myth #4: Liquid Clustering doesn’t work at petabyte scale.

Databricks has significantly improved OPTIMIZE operations. Dozens of customers now run petabyte-scale Liquid Clustered tables in production, with planning time reduced from hours to minutes.

Myth #5: Liquid Clustering only benefits Databricks readers.

Liquid Clustering is a write-side optimization that produces standard Parquet files with min/max stats. Any compatible reader, including open-source Spark and DuckDB, can leverage these stats for efficient data skipping.

Myth #6: Partitioning is necessary for concurrent ETL.

This is a workaround for older concurrency models. Liquid Clustering provides row-level concurrency, allowing multiple writers to update different rows within the same file without conflict. This removes a primary reason teams relied on partitioning for write boundaries.

Myth #7: Z-Ordering compensates for partitioning’s weaknesses.

Z-Ordering suffers from poor clustering quality and requires frequent, costly rewrites to maintain effectiveness. Liquid Clustering, however, incrementally clusters data, including at write time, keeping the layout optimal without unnecessary data churn.

Myth #8: Partitioning is required for selective data overwrites.

Databricks supports selective overwrites natively on Liquid tables using REPLACE USING and REPLACE ON syntaxes, which are atomic and work across any compute environment, unlike Dynamic Partition Overwrites.

Success Stories

Migrating to Liquid Clustering has yielded significant results. Arctic Wolf saw a 7.7x query speedup on a 3.8 PB security telemetry table, reducing query times from 51 seconds to 6.6 seconds and dropping file count by half. Bolt observed a 138% increase in write throughput and up to a 63% reduction in read times on critical CDC tables during an in-place conversion with zero downtime.

The advantages of Databricks Liquid Clustering are clear, offering a more efficient and flexible approach to data management for modern analytical workloads.

This advancement is also a boon for business intelligence, potentially leading to faster insights and reduced costs, as seen in discussions about the Databricks BI Stack.