Artificial Intelligence

Databricks makes serverless move a review step

Databricks launched Genie Migrate for Serverless in beta to turn classic-to-serverless moves into reviewable diffs with rollback.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

Databricks Genie Migrate for Serverless migration review flow
Credit: Databricks

Databricks launched Genie Migrate for Serverless in beta to move classic jobs and notebooks to serverless compute. Serverless compute is Databricks execution where Databricks handles provisioning, scaling and optimization so teams run jobs without managing clusters. Genie Migrate for Serverless scans a fleet, proposes config and code fixes as diffs, and applies them reversibly with a 30-day rollback, Databricks says.

Will Genie Migrate for Serverless rewrite production code on its own?

No, Genie Migrate for Serverless shows every proposed edit as a side-by-side diff before anything is written, Databricks says. Data engineers who own classic jobs keep approval control and judgment on whether a migrated run looks healthy. The flow runs from the /compute migrate command in Genie on any job or notebook with no setup because Databricks manages the agent inside the workspace. Assess marks what can move, translate maps cluster and library configs, and apply validates then clones.

The hard part is translation that used to live in trial and error. A classic job fails on serverless for access mode, a Spark config that does not apply, or a library that resolves on classic but not on serverless. Genie Migrate for Serverless carries over confs that still apply and surfaces uncertain ones for you to decide instead of deleting them silently, Databricks says. Spark Connect is the remote interface serverless uses to separate client code from cluster execution. Where code uses incompatible APIs, Genie Migrate for Serverless rewrites the pattern and shows the diff, Databricks says.

Bulk work shifts to the Databricks CLI using the compute-migrations command group, which builds serverless working copies without an open Genie session, Databricks says. Bulk migration needs CLI version 1.24.0 or above, covers jobs today with notebook support coming soon, and keeps job ID, schedule and permissions on approval, Databricks says. Jobs managed by Databricks Asset Bundles, which is the infrastructure-as-code format for packaging jobs and configs, or notebooks in a Git folder cannot migrate yet.

Legacy workspaces without Unity Catalog, which is the Databricks governance layer for data and access control, lack access to serverless until they upgrade, per Databricks serverless docs. Genie Migrate for Serverless performs deterministic checks today and keeps the original intact for revert, but sandbox validation that would compare a sandboxed run against the classic baseline is still coming, Databricks says. Testing on production tables can still write data that a revert will not undo. Start with a noncritical job, run /compute migrate, review the what-will-change panels, and watch the first serverless run before moving the fleet.

Mentioned in this story

Sources

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsDatabricksGenie Migrate for ServerlessGenieUnity CatalogSpark ConnectDatabricks Asset BundlesSpark

StartupHub.ai StaffDaniel SingerWritten by StartupHub.ai Staff, Editorial teamReviewed by Daniel Singer, EditorAbout

StartupHub.ai Staff, Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer, Editor

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

More from Daniel Singer

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.