Backstage Ditches Postgres for Databricks Lakebase

The long-standing divide between operational and analytical databases is finally dissolving. Databricks is demonstrating this shift by integrating Backstage, Spotify's internal developer portal, with its Lakebase. This move effectively moves Backstage off its traditional PostgreSQL database and onto Databricks' unified data platform.

Historically, operational (OLTP) and analytical (OLAP) databases required separate infrastructure, budgets, and on-call rotations due to fundamental differences in storage, compute, and failure modes. However, with shared storage, serverless, workload-isolated compute, and catalog-level governance, these distinctions are becoming mere access patterns on a single foundation.

The Setup: Pointing Backstage at Lakebase

Lakebase now offers a serverless PostgreSQL surface, powered by Neon's architecture, directly within the Databricks Platform. Because it speaks the wire-protocol PostgreSQL, Backstage operates seamlessly, unaware it's not connected to a standard RDS instance. The integration involved updating Backstage's configuration to point to Lakebase and replacing its default in-memory search with PgSearchEngine.

A key hurdle was authentication. Lakebase requires OAuth JWTs instead of traditional Databricks Personal Access Tokens. The `databricks postgres generate-database-credential` CLI command provides short-lived, scoped JWTs, suitable for applications and CI/CD. For this proof-of-concept, a cron script refreshed these credentials every 50 minutes to manage expiration.

With authentication resolved, Backstage's Knex migrations ran without issue, bringing the portal online.

Branching Changes the Database Development Cycle

The true innovation lies in how this integration transforms the database development lifecycle. Traditional PostgreSQL's tempo often forces teams into slow, expensive processes for tasks like schema migration testing.

Making a copy of a traditional database for testing is slow and costly, leading teams to skip thorough validation and rely on maintenance windows. The ability to create near-instant database branches fundamentally changes this. With Lakebase's copy-on-write architecture, branching creates a pointer to existing data, diverging only on write, making the operation instantaneous.

This capability allows developers to ask not "is this change safe enough?" but "which copy of production should I test this on first?" This aligns with the concept that Databricks Postgres branches like Git, enabling rapid iteration.

A minor gotcha: the API requires all parameters to be nested within a 'spec' object and a time-to-live (ttl), expire_time, or no_expiry must be specified. The control plane acknowledged the request instantly, and a ~63 MB clone of the Backstage catalog landed in just over a second.

Point-in-Time Recovery: The Undo Button

Branching and Point-in-Time Recovery (PITR) are fundamentally the same primitive. To test recovery, the 'final_entities' table was wiped. A recovery branch was created from a timestamp seconds before the deletion.

The entire recovery process took less than four seconds. Verifying the data confirmed all 32 entities were restored, while production remained at zero, proving the isolation of branches. Notably, Lakebase snapped to the nearest WAL record, 12 seconds prior to the requested timestamp, offering WAL-level granularity for time-sensitive recovery, crucial for understanding Point-in-Time Recovery database operations.

When database state becomes a cheap, forkable artifact, risky operations gain a dry run, and incidents get an instant undo button.

From Infrastructure Capability to Developer Workflow

The technical feasibility of database branching is proven: instant clones and rapid recovery with a real application unaware of the underlying change. The challenge now is integrating this into the daily developer workflow, making it as natural as branching code.

This is where significant gains in developer productivity emerge. The goal is to make database branching invisible, automatic, rather than an explicit command. Work is underway on VS Code extensions to synchronize Git and database branches automatically.

The traditional sprint cycle involves creating Git branches, writing extensive mock objects for database interfaces, and testing against mocked or in-memory databases. This often leads to discovering schema migration issues only after deployment to staging environments.

With database branching, the developer workflow transforms: Git branches trigger automatic, sub-second Lakebase database branches. Developers write and test code against live database data from the outset. Integration tests run against the actual database, eliminating the need for mocks.

Multiple solutions can be experimented with, and rollbacks are trivial. Pull requests trigger CI to create its own database branch, validating both code and schema. QA teams gain disposable branches for destructive testing. Merging a PR allows CD pipelines to migrate changes upstream and clean up branches, code and data.

Mock objects, which can constitute 20-30% of test code, become obsolete. Staging environment collisions vanish. The common "works on my machine" problem disappears. Database changes discovered during development are cheap to fix.

Instant branches enable performance tests, disposable branches facilitate functional tests, and dedicated running branches streamline User Acceptance Testing (UAT).

The costly infrastructure built around a now-obsolete constraint, slow, expensive database copies, is no longer necessary.

The question for development teams is how much of their sprint is spent on workarounds for a constraint that no longer exists.

Part 2 of this series will explore the implications for security and compliance when operational databases are absorbed into Databricks' Unity Catalog.