World Bank Leverages Databricks for Poverty Fight

The World Bank Group is tackling global poverty with a new unified data and AI platform built on Databricks. The initiative aims to transform vast datasets into actionable insights, accelerating efforts to improve shared prosperity worldwide.

Historically, the organization faced challenges integrating disparate structured operational data with millions of unstructured documents. Researchers and analysts spent considerable time manually sifting through libraries to answer fundamental questions about past projects, creating significant bottlenecks.

Unifying Knowledge, Eradicating Bottlenecks

Databricks' Lakehouse Architecture provided the foundation. The platform integrates structured data with unstructured content, enabling faster, more informed decisions. This approach eliminates manual research, a critical step in driving poverty reduction efforts.

A key component, Unity Catalog, offered a unified interface for data governance. Databricks Volumes allowed for scalable management of unstructured documents alongside structured data.

For business users, Databricks Genie provided natural language access to data, removing the need for SQL expertise. The Databricks AI Gateway ensures centralized control over agent access, costs, and security.

The World Bank Group's journey began with migrating operational data and establishing governance. This laid the groundwork for an outcomes-driven corporate scorecard.

When structured queries showed inconsistencies, a metrics layer was implemented to ensure deterministic answers, crucial for financial and operational reporting.

The organization then focused on unstructured content, using Databricks Volumes and vector search for retrieval-augmented generation. This enabled natural language queries against project documents, vastly improving search efficiency.

To handle complex, multi-domain questions, an agentic layer was developed. This system uses an intent classifier, domain classifier, and query decomposer to route requests efficiently, assembling results from multiple specialized agents.

This architecture is not unlike traditional multi-tier web design, but updated for an AI context.

External feedback sessions with NGOs and government representatives across Africa and East Asia Pacific refined the system, ensuring it met real-world needs.

Accelerating Impact

The platform now supports three million document downloads monthly, with half originating from low- and middle-income countries. A prototype spanning multiple regions was deployed in just two and a half days, a stark contrast to previous multi-year project timelines.

This initiative is central to the World Bank Group's flagship Knowledge 360 and Data 360 projects, aiming to make knowledge accessible across its institutions and to all stakeholders.