The fragmentation of modern medicine's vast multimodal data across siloed systems has historically prevented a holistic understanding of patient care. Existing models fail to capture the full temporal depth of clinical records. Addressing this, researchers introduce the Apollo multimodal temporal foundation model, a significant leap towards unified patient representation.
Bridging Siloed Data into a Unified Patient Atlas
Trained on over three decades of longitudinal hospital records from a major US hospital system, Apollo integrates 25 billion records from 7.2 million patients. It unifies 28 distinct medical modalities, including structured data, clinical text, and images, across 12 major medical specialties. This creates an 'atlas of medical concepts' encompassing over 100,000 unique medical events, forming a computational substrate capable of modeling entire patient care journeys. These journeys, comprised of sequences of structured and unstructured events, are compressed by Apollo into virtual patient representations, as detailed in their publication on arXiv.
Forecasting Patient Trajectories with Unprecedented Breadth
The potential of these whole-patient representations is validated through 322 prognosis and retrieval tasks on a held-out set of 1.4 million patients. The Apollo multimodal temporal foundation model demonstrates generalized clinical forecasting capabilities. These include predicting new disease onset risk up to five years in advance (95 tasks), disease progression (78 tasks), treatment response (59 tasks), risk of treatment-related adverse events (17 tasks), and hospital operations endpoints (12 tasks). Feature attribution confirms that predictions align with clinically interpretable multimodal biomarkers, adding a layer of trust and understanding to its forecasting power.
Enabling Computable Medicine Through Multimodal Search
Beyond predictive tasks, Apollo excels in semantic similarity search across 61 retrieval tasks. It functions as a multimodal medical search engine, capable of handling text and image queries. This multimodal search capability, combined with its predictive prowess, establishes the foundation for 'computable medicine', a future where the full context of patient care is accessible for sophisticated computational reasoning.