ClinEnv: Bridging LLM Gaps in Clinical Decision-Making

The complexity of clinical practice, characterized by incremental information gathering, sequential irreversible decisions, and inherent uncertainty, remains a significant challenge for AI evaluation. Existing benchmarks fall short, often compromising on critical aspects of this dynamic process. To address this, researchers have introduced ClinEnv, an interactive benchmark designed to simulate real inpatient admissions and rigorously assess Large Language Models (LLMs) as attending physicians.

Simulating the Physician's Sequential, Uncertain Workflow

ClinEnv moves beyond static evaluations by constructing each medical case into an ordered sequence of decision stages. At every stage, LLMs must actively query specialized agents to gather heterogeneous information before committing to crucial decisions like medications, procedures, and diagnoses. This paradigm, termed Longitudinal Inpatient Simulation, mirrors the actual, step-by-step nature of clinical reasoning, providing a far more realistic assessment environment than static datasets.

Quantifying Information Acquisition and Decision Quality

The benchmark meticulously scores both the final decisions made by the LLM and the quality of the information-gathering process itself. Through deterministic ontology-grounded matching, ClinEnv provides concrete metrics for decision accuracy. Crucially, it makes the information-acquisition gap, often invisible in outcome-only evaluations, directly measurable. Across seven evaluated models, the strongest performer achieved only a 0.31 decision F1 score, with outcome quality sharply decoupled from process quality. A notable concentration of difficulty was observed in management decisions and later stages of patient care, where models reliably recovered discharge diagnoses (0.51 F1) but struggled significantly with management actions (0.17 F1) and continued to issue redundant queries.

The Criticality of Process Evaluation in Medical AI

The findings from the ClinEnv benchmark underscore a critical insight: evaluating LLMs in complex, sequential domains like medicine requires more than just assessing final outcomes. The tendency for models to recover diagnoses more reliably than management actions, coupled with inefficient information-seeking behavior, highlights a fundamental gap in their practical applicability. This information-acquisition deficit, made visible by the ClinEnv benchmark, is a crucial area for future AI research and development, particularly for applications demanding high-stakes, dynamic decision-making.