Deutsche Börse uses GenAI for data migration

Deutsche Börse Group is leveraging generative AI to tackle the monumental task of migrating thousands of analytics notebooks. Facing a 2027 deadline to move from Cloudera's soon-to-be-decommissioned Zeppelin notebooks to Databricks, the financial services giant developed a custom Databricks App to automate the process.

The challenge involved over 2,000 users and a vast number of notebooks, many containing complex logic and institutional knowledge. A manual rewrite would have taken years, prompting the development of an AI-assisted solution. The core innovation, as detailed on the Databricks blog, lies in separating structural conversion from logic reconstruction.

Automating the Structure, AI for Logic

The Databricks App handles the deterministic aspects: transforming Zeppelin's paragraph format into Databricks cells, translating interpreter syntax, and reformatting metadata. This structural conversion preserves the original content precisely.

The true power comes in the next step: logic reconstruction. For each converted notebook, the app generates a context-aware prompt tailored to Deutsche Börse's specific Zeppelin environment, including custom interpreters and data source references. This prompt is fed into Databricks Genie, an AI assistant that asks clarifying questions and rebuilds the notebook's logic.

This hybrid approach, automating the predictable and delegating the complex to AI, is key to the success of this Databricks generative AI migration.

From Hours to Minutes

The result is a dramatic reduction in migration time. Notebook redevelopment that once took hours now takes a mere 15-20 minutes per notebook. This significantly accelerates a critical cloud transformation initiative.

Crucially, the tool democratizes the migration process, requiring less deep technical expertise from business users. The Databricks Streamlines Lakehouse Migrations initiative is exemplified by this project, showcasing how platform capabilities can be combined with AI for efficiency.

Lessons Learned

Deutsche Börse highlighted several key takeaways: avoiding over-engineering, recognizing the limitations of rule-based systems for heterogeneous content, and the critical importance of context in AI prompts. Early collaboration with the platform team was also vital.

This project demonstrates that AI-assisted migration is not a future concept but a present reality, transforming daunting cloud migration challenges into manageable, scalable workflows.