Databricks tags dbt pipelines

Databricks is rolling out Query Tags, a new feature designed to bring much-needed clarity to the often opaque world of data pipeline costs. This enhancement, detailed in an announcement from Databricks, promises granular usage attribution for dbt pipelines, allowing teams to track precisely where compute resources are being consumed.

For too long, understanding the cost implications of complex dbt projects has been a significant challenge. When a warehouse bill doubles, pinpointing the exact models or teams responsible can feel like searching for a needle in a haystack, especially when query histories show little more than generic labels like 'Databricks Dbt.' Query Tags aim to solve this by automatically injecting metadata and enabling custom tagging for every query generated by a dbt pipeline.

Automated Insights, Custom Control

The integration with the dbt on Databricks adapter (version 1.11 and above) offers multiple layers of tagging. Databricks automatically injects tags like the dbt model name, materialization strategy, and adapter versions. This out-of-the-box visibility requires zero configuration.

Beyond these automated tags, users can define profile-level tags within their dbt profiles. This allows for consistent tagging across an entire project, specifying dimensions like team, cost center, project name, and environment. For even finer control, model-level tags can be applied directly within dbt_project.yml or SQL model definitions, overriding profile-level tags if conflicts arise.

All these tags are recorded in system.query.history, transforming raw query logs into actionable data.

From Raw Data to Actionable Analytics

With tags populated in the query_tags column (a MAP), users can easily query their data warehouse to understand resource consumption. This directly addresses the challenge of cost attribution, eliminating the need for manual log analysis or resource splitting.

Databricks' new Databricks Adds Query Context feature, which leverages these tags, allows users to ask plain-language questions via Genie or write SQL queries for repeatable analysis. This provides immediate insights into which dbt models are the most resource-intensive.

The reference project includes a self-monitoring dashboard that analyzes its own billing data. This dashboard visualizes key metrics such as total compute time per model, materialization splits, and daily activity, offering a clear picture of pipeline performance and cost distribution.

Tagging metric views, a newer materialization type in dbt-databricks, is also supported, allowing for specific tracking of queries related to these objects.

Best Practices for FinOps

Databricks recommends a consistent tagging hierarchy, prioritizing profile-level tags for organizational context (team, cost center) and reserving model-level tags for exceptions. Always tag the environment (e.g., local-dev, dev, staging, prod) to distinguish between development and production runs.

Using `project_name` is crucial when multiple dbt projects share a warehouse, enabling cost attribution per pipeline. Custom tags should focus on business context that dbt cannot infer, such as ownership or project identity, avoiding duplication of auto-injected information.

The complete reference project, demonstrating Query Tags end-to-end, is available on GitHub, allowing users to clone, deploy, and adapt it to their own dbt projects.