Data transformation tool dbt is finding a more powerful home on the Databricks Lakehouse. The combination promises to streamline data workflows by embedding dbt into a unified platform, moving away from the fragmented approach common in many data stacks. This integration aims to tackle issues like data duplication, inconsistent permissions, and complex observability that plague multi-system architectures.
Companies working on this
Profiles of the companies named in this story, with founding year, headquarters, and a short description from our database.
A unified data analytics and AI platform built on the lakehouse architecture.
- Founded
- 2013
- Location
- San Francisco, United States
- Valuation
- $190.0B
The appeal of running dbt on Databricks lies in its ability to deliver on four key pillars: open foundations, seamless orchestration, integrated governance, and strong price-performance. This approach directly addresses the limitations of proprietary systems that often lead to vendor lock-in and increased operational friction.
Open Foundations for Data Portability
Vendor lock-in remains a significant concern for data strategies. While dbt itself is built on an open adapter framework, its effectiveness is tied to the underlying data platform. Databricks promotes an open lakehouse architecture, utilizing open table formats like Delta Lake and Apache Iceberg. This ensures transformed data remains accessible across various tools and environments, not confined to a single query engine. This openness extends to Unity Catalog, which supports governed access from external engines, and Databricks SQL, adhering to ANSI standards for query portability.
Unified Orchestration with Lakeflow Jobs
Operational complexity often arises from managing separate orchestration tools alongside data platforms. Lakeflow Jobs on Databricks integrates dbt as a first-class task type. This allows teams to orchestrate dbt transformations alongside data ingestion and downstream actions within a single workflow. Failures, retries, and job context are visible in one place, eliminating the need to switch between disparate systems for debugging.
Governance Embedded by Default
As dbt workflows scale, governance becomes critical. Unity Catalog unifies access control, discovery, and lineage for the entire lakehouse, extending beyond dbt to ingestion, BI, and ML/AI applications. Permissions are managed at the schema level and persist across table rebuilds, simplifying administration. Fine-grained controls like row-level filters and column masks apply consistently across dbt, BI tools, and notebooks.