Databricks Powers Global Health Volunteer Matching

Databricks is powering a critical initiative by the Virtue Foundation to connect medical volunteers with essential health services in 72 countries. This collaboration leverages AI to build a comprehensive, actionable database of global healthcare infrastructure, addressing critical gaps in underserved regions. The project exemplifies how advanced data analytics can drive significant humanitarian impact.

The Virtue Foundation, a nonprofit dedicated to improving global health delivery, operates VF Match, a platform connecting medical professionals to volunteer opportunities. Databricks for Good has been instrumental since 2024, applying AI to aggregate and analyze data from numerous low and low-middle income countries. Initial proofs of concept demonstrated the power of Large Language Models (LLMs) in extracting structured data from web sources, mapping healthcare facilities, and identifying service deficits.

The initial proof of concept has since evolved into a robust, production-grade system hosted on Databricks. This platform aggregates data from thousands of healthcare facilities and non-profits worldwide, transforming disparate information into a unified, accessible format. This upgrade significantly enhances the Virtue Foundation's ability to match skilled medical volunteers with the most pressing needs.

Building the Foundation: Global Healthcare Data at Scale

At the core of the initiative is the Foundational Data Refresh (FDR), a comprehensive dataset built from various web-based sources. This refresh systematically ingests and updates information from 72 countries, drawing on open-source geospatial data from Overture Maps and real-time web scraping via Bright Data.

The data extraction pipeline relies on OpenAI’s GPT models, processed efficiently using Databricks and Apache Spark. To handle the scale and complexity, the pipeline breaks down extraction into targeted steps: classifying medical relevance, identifying organization types, and extracting specific services and specialties. This methodical approach minimizes token usage and maximizes precision.

Key features ensure the pipeline's scalability and production readiness. Extensible data modeling uses a star schema for simplified analytics and faster queries. Status-based checkpointing allows pipelines to resume without costly reprocessing of LLM calls. A configurable extraction registry modularizes logic, and scalable distributed processing handles multi-terabyte workloads using Spark and Photon for high performance. Lakeflow Jobs orchestrate over a dozen interdependent tasks with sophisticated retry policies.

Entity Resolution for Data Integrity

A significant challenge is entity resolution, ensuring that duplicate records from various sources are unified. Messy data with inconsistent names and addresses often breaks traditional deduplication methods. The project employs Splink, an open-source probabilistic record linkage framework, to create a single, authoritative record for each facility and NGO.

Running probabilistic matching at scale revealed performance bottlenecks. Pairwise comparisons create inherently skewed workloads, leading to significant delays. Enabling Databricks' vectorized query engine, Photon, reduced worst-case data partition processing times by 15x, from 30 minutes to approximately 2 minutes.

The VF Agent: Natural Language Meets Healthcare Data

Looking ahead, a prototype agent has been developed to enable experts to query data using natural language. This multi-agent architecture, built with LangGraph, utilizes Databricks Model Serving, Vector Search, and Genie. The system translates user queries into standardized medical terminology, routing them to specialized agents for facility discovery or analytical queries against structured data.

Ultimately, healthcare professionals can now more rapidly discover up-to-date volunteer opportunities and access global data on thousands of facilities. The journey from proof of concept to a production system on Databricks highlights the potential of AI in addressing critical global health challenges.