Databricks Connects Dev Ops and FinOps

Databricks is pushing for a unified data plane that bridges the gap between development operations and financial operations. The latest demonstration, detailed in the third part of its Backstage on Databricks series, shows how its Lakebase architecture can bring together disparate data sources for unprecedented visibility into cloud spend and infrastructure ownership. This moves beyond traditional ETL pipelines, offering real-time insights directly within a single SQL query. The goal is to dissolve the long-standing separation between operational and analytical data worlds.

In a typical enterprise setup, tracking infrastructure costs and ownership is a fragmented process. Ownership details often reside in developer tools like Backstage, while cost data lives in a separate data warehouse. Linking these two requires manual intervention, ETL jobs, or cumbersome communication chains. Databricks posits that its Lakehouse architecture, specifically with Lakebase, can eliminate these friction points. The core innovation lies in separated compute, allowing live portal use and massive analytical queries against the same data substrate without performance degradation. Backstage, for instance, operates within its own compute envelope, ensuring its rapid portal queries don't interfere with FinOps analyses.

The FinOps Payoff

The practical application of this unified approach is demonstrated through a FinOps join. By using Databricks Lakehouse Federation, a FinOps analyst can query operational data directly from Backstage’s Postgres database and join it with Databricks’ own billing data within Unity Catalog. This eliminates the need for data movement. The system pulls resource names directly from the live operational Postgres catalog and merges them with system billing rows, a feat previously requiring separate data silos and complex integration.

This approach bypasses the inefficiencies of traditional ETL. Skeptics might question why a simple Python script syncing data periodically wouldn't suffice. The answer, according to Databricks, lies in the dynamic nature of modern development. When developers create ephemeral database clones for testing pull requests, setting up new ETL pipelines for each temporary environment becomes impractical. Lakebase, however, makes this data instantly queryable the moment a branch is created. In a proof-of-concept, a dropped test branch was automatically attributed with a cost of 0.0107 DBU.

Operationalizing Convergence

The convergence of operational and analytical data is technically proven, but operationalizing it presents practical challenges. A key hurdle is the current authentication mechanism for Lakehouse Federation’s Postgres connector, which supports only static user/password credentials. Since Lakebase authenticates app identities via OAuth JWTs, a parallel authentication path must be established. The workaround involves creating a separate, native Postgres role with SCRAM-SHA-256 authentication for federation, distinct from the app’s OAuth identity. This requires managing two authentication paths and handling password rotation, though Databricks suggests this can be scripted for repeatable setup.

The potential for native OAuth JWT support in Federation would eliminate this workaround entirely. Beyond infrastructure cost allocation, this unified data access offers granular cost visibility for development teams. Engineering managers can analyze the compute costs associated with development branches during sprints, comparing dev/test expenses against production. This data allows for informed decisions on branch lifecycle policies, ensuring that ephemeral branches are treated as such in billing data to prevent orphaned resources from accumulating costs. The 0.0107 DBU cost for a test branch is presented not as an expense, but as a highly measurable productivity investment, enabling conversations about database costs that were previously impossible.

What Comes Next

The Lakebase endpoints scale to zero when inactive, meaning compute costs are only incurred when branches are actively queried. This prevents idle branches from incurring charges. Databricks and its partner Thoughtworks are working to move this from a proof-of-concept to production-ready implementation. The article concludes by urging teams to embrace this shift, suggesting practical steps like branching migrations on real schemas or rewriting mock-heavy test suites against live branches, and joining billing data to ownership graphs. The teams that adopt these practices first will shape the future of data management and cost transparency.