Databricks Adds Query Context

Databricks is introducing a new feature called Query Tags, aiming to provide crucial context to data warehouse operations that was previously missing. This feature allows users to attach custom key-value pairs to SQL executions, offering a granular way to track and attribute workloads.

The core problem Query Tags addresses is the lack of detail in standard query logs. While Databricks SQL logs essential information like who ran a query and from which tool, it often falls short when it comes to attributing costs or pinpointing specific issues. For instance, knowing a slow query originated from Power BI is helpful, but identifying the exact dashboard causing the slowdown requires more context.

Attributing Costs and Tracing Workloads

Query Tags enable users to segment shared warehouse costs by team, project, or dashboard, moving beyond simple user-based attribution. This is particularly beneficial for chargeback models and financial accountability.

The feature also significantly enhances the ability to trace queries originating from third-party tools. Databricks has integrated automatic tagging for popular platforms like dbt, Power BI, and Tableau. For dbt, every query is automatically tagged with model names, core versions, adapter versions, and materialization types, simplifying the identification of performance regressions. ASOS engineering leads Dipesh Bhundia and Dave Couse noted that this allows them to "accurately split up warehouse costs by the teams that are running dbt on it."

Power BI and Tableau support custom tags at the connection level, meaning once configured, all subsequent queries from that connection inherit the tags automatically. This ensures that even if a workbook is renamed in Tableau, its attribution is preserved.

Context for Custom Applications and Ad-Hoc Work

Beyond partner tools, Query Tags are valuable for custom applications built on Databricks. Developers can attach metadata such as `customerid`, `applicationname`, or `app_version` directly to queries executed via APIs or connectors. This transforms what would otherwise be anonymous API calls into traceable workloads.

Unit21 DevOps Engineer Matthew Haber shared how Query Tags helped them maintain visibility after consolidating warehouses: "With Query Tags, we just pass the team name from our Databricks SQL Connector for Python workloads and we have that attribution back, no need to split warehouses again."

Analysts performing ad-hoc work also benefit. Queries run in the SQL Editor, Notebooks, or Dashboards can be tagged with dimensions like 'dev vs. prod', 'cost center', or 'experiment name', making it easier to manage and analyze diverse workloads without annotating every single query individually.

Monitoring and Analysis with System Tables

Once queries are tagged, this metadata is stored in the `query_tags` column of the Query History System Table. This opens up powerful analytical possibilities using standard SQL or natural language queries via Genie. Users can now easily answer complex questions like which dbt model introduced a regression or which team is driving warehouse costs. Databricks is also working on bringing searchability to the Query History UI and extending Query Tags to Serverless Notebooks and Jobs.

Query Tags are currently in Public Preview for SQL Warehouses.