Databricks Auto Upgrades debut

Databricks is rolling out a new feature designed to streamline the adoption of cutting-edge lakehouse table capabilities. Dubbed Databricks Auto Upgrades, this system aims to bring best-practice features to Unity Catalog (UC) managed tables with minimal user intervention, as detailed on the Databricks blog.

The core challenge addressed by Auto Upgrades is the manual overhead typically associated with implementing new table features. Historically, adopting these advancements required identifying eligible tables, verifying client compatibility, and executing manual commands, a process often too time-consuming for data teams. This new capability promises to automate that effort, improving performance, reliability, interoperability, and cost efficiency.

Automating Lakehouse Evolution

Auto Upgrades operates by observing table access patterns over a rolling 100-day window. It then verifies that all Databricks clients accessing the table support the feature and that the table is actively being used. Only after these strict conditions are met does it safely apply the feature via a background job.

This automated approach offers a more thorough update process than manual methods.

• Features are only enabled if they are generally available and do not negatively impact performance or costs.
• The extensive observation window captures infrequent workloads, ensuring broad compatibility.
• Strict verification ensures every accessing client supports the feature before it's applied.
• The system avoids upgrading tables it cannot fully verify, such as those with external client access (though future support is planned).
• All enabled features can be disabled or dropped on a per-table basis, preserving user control.

Unlocking Key Lakehouse Benefits

As Auto Upgrades runs, tables gain access to a suite of best-practice features that enhance their functionality.

These include optimizations like Automatic Liquid Clustering for improved data layout, Deletion Vectors for more efficient updates and deletes, and Column Mapping for instant schema changes without data rewriting. Parquet V2 compression also contributes to lower storage costs and faster scans.

Interoperability is enhanced through Catalog Commits, enabling cross-engine access and governance for UC managed tables. Row Tracking introduces row-level identifiers, paving the way for features like Automatic Change Data Feed and incremental Materialized View refreshes.

Reliability is bolstered by Checkpoint V2, which provides a more scalable format for table metadata, reducing commit failures under heavy write loads.

Observability and Getting Started

Databricks Auto Upgrades ensures visibility into all changes. Each upgrade is logged in the table's DESCRIBE HISTORY output and Catalog Explorer, distinctly marked from user-initiated actions. A system table will offer account-wide visibility into all Auto Upgrades events.

The feature currently applies to Unity Catalog managed tables. Users are encouraged to convert their existing tables to this format to benefit from Auto Upgrades.

Databricks Auto Upgrades aims to keep lakehouse tables current without manual effort.

The system is designed to be non-disruptive, avoiding tables with unsupported clients or infrequent usage.

This initiative represents a significant step toward a more self-managing lakehouse, allowing users to focus on insights rather than infrastructure maintenance.