Databricks Adds dbt Query Tagging

Databricks is enhancing its data and AI platform by integrating granular usage attribution for dbt pipelines through a feature called Query Tags. This development, detailed in a recent Databricks blog post, allows data teams to tag every dbt query with specific metadata, simplifying cost management and performance analysis.

Previously, understanding the exact cost or compute time associated with individual dbt models could be challenging. When a dbt project ran dozens or hundreds of models, query history often showed generic labels like 'Databricks Dbt.', making it difficult to pinpoint resource consumption. Query Tags, now in public preview, aim to solve this by automatically injecting metadata such as the dbt model name and materialization strategy. Users can also add custom tags, like team, cost center, or environment, to further refine attribution.

How Query Tags Integrate with dbt

The dbt-databricks adapter, version 1.11 and later, natively supports Query Tags. These tags can be applied at three levels. First, auto-injected tags provide immediate visibility into model execution details without any configuration. Second, profile-level tags can be added to a dbt profile's configuration, applying to all queries run under that profile. This is an efficient way to tag entire projects with dimensions like team, cost center, project name, and environment. Third, model-level tags offer even finer control, allowing specific tags to be defined for individual dbt models within the `dbt_project.yml` file or directly in the model's SQL definition. Model-level tags merge with profile-level tags, with model-specific values taking precedence.

All these tags are recorded in the `system.query.history` table, accessible via SQL. This means data teams can query this table to see exactly which dbt models are consuming the most warehouse resources or compute time. The information is available through standard map access syntax, allowing for aggregation into reports or dashboards.

Driving FinOps and Performance Insights

The primary benefit of Query Tags is enabling more effective FinOps practices. By attributing costs directly to specific dbt models, teams can identify usage hotspots and optimize resource allocation. For instance, a reference project provided by Databricks demonstrated that four specific mart tables accounted for 92% of compute time, an insight previously obscured. This granularity helps answer critical questions, such as which team is responsible for increased warehouse bills or where optimization efforts should be focused.

Beyond cost attribution, Query Tags are invaluable for performance debugging and workload monitoring. Teams can quickly identify slow-running models or understand the impact of different materialization strategies (table, view, incremental, metric_view) on query performance. Databricks also offers a reference project that includes an AI/BI dashboard. This dashboard queries `system.query.history` filtered by the project's own query tags, effectively creating a self-monitoring system that tracks its own costs and performance metrics.

Tagging Metric Views and Best Practices

Databricks' new metric views, a materialization type that defines reusable business semantics in Unity Catalog, can also be tagged using the `query_tags` configuration parameter. This distinguishes them from `databricks_tags`, which are Unity Catalog object tags for governance and discovery. Query tags on metric views track the queries used to create or refresh them, providing performance context for these semantic definitions.

Databricks offers several best practices for implementing Query Tags. They recommend using a consistent tag hierarchy, defining organization-wide tags at the profile level (team, cost_center, project_name, env), and reserving model-level tags for exceptional cases to avoid configuration sprawl. Always tagging the environment (e.g., local-dev, dev, staging, prod) is crucial for separating development queries from production runs. Using `project_name` is advised when multiple dbt projects share a warehouse, allowing costs to be attributed per pipeline. Finally, teams are advised not to over-tag, focusing custom tags on business context that dbt cannot infer, such as team ownership or cost center, rather than duplicating auto-injected metadata.

The complete reference project, including the dbt pipeline, analytics dashboard, and deployment configurations, is available on GitHub. This initiative aligns with the broader trend of increasing transparency and control over cloud data warehouse spending, a significant concern for many startups and enterprises alike. Companies like Snowflake (NASDAQ:SNOW), a key competitor in the data warehousing space, also offer various cost management and optimization tools, highlighting the competitive focus on efficiency and observability in the cloud data market.