Hinge Health's Rashi Agrawal on Healthcare AI Guardrails

Rashi Agrawal, who leads AI and ML at Hinge Health, recently shared critical insights into building safe and reliable member-facing healthcare AI. Speaking at the AI Engineer World's Fair, Agrawal emphasized the paramount importance of robust guardrails, particularly in a sector where errors can have life-threatening consequences.

The State of Healthcare AI and Its Perils

Agrawal opened by highlighting the current state of healthcare AI, noting that while approximately 40 million people use these models for triaging health issues, significant risks remain. She cited alarming headlines: a chatbot advising a user to ingest sodium bromide instead of cutting salt, leading to hospitalization, and a Mount Sinai study finding a health AI under-triaged life-threatening emergencies 50% of the time. The ECRI, a patient safety group, even named AI chatbot misuse as the number one health technology hazard for 2026.

These examples underscore that the challenges are not merely theoretical but are present in production systems. The core question Agrawal addressed is: "How do you ship AI to somebody who has already trusted you with their health?" Her answer centers on three non-negotiable foundations:

• The constraint is the architecture. Most AI safety failures stem from architectural decisions made before any code is generated.
• Deterministic rules belong above the model, not inside it. What cannot be wrong should not be left to probability.
• Safety is a continuous evaluation layer, not a one-time gate. The launch of a product is the beginning of risk, not the end.

Layer 1: Architecture as the Foundation for Safety

Agrawal stressed that protecting Protected Health Information (PHI) requires both policy and architecture. The key shift for member-facing health AI is how PHI is handled. Instead of treating it as a runtime problem to be redacted reactively, she advocates for stripping PHI at the pipeline boundary, before it ever enters the data lake. This ensures that even when developers access dashboards, the sensitive data is already gone.

Furthermore, strict isolation between production and non-production environments is crucial, as even a single link can lead to data leaks. Compliance rules like HIPAA, FDA's Good Machine Learning Practice, and state laws must be foundational inputs in system design, not afterthoughts. Agrawal stated, "You cannot slap on HIPAA on top of, you know, an underlying system or an architecture. You start with it and let the architecture grow around it."

Layer 2: Deterministic Code Above Probabilistic Models

Probabilistic systems excel at generation but are unreliable for tasks where errors are unacceptable. Agrawal's core principle here is: "Must not fail behavior belongs above your prompt, above the model." This translates to a code layer that runs before the model, making irreversible decisions.

This code layer is responsible for critical actions such as emergency escalations (e.g., routing to 911 or 988 if self-harm is mentioned, without the model even seeing the input), intent routing to the correct agent, and identity verification. She cautioned that even a system prompt is not a security boundary, as it can be overridden. "A model is not a guardrail. A model with a system prompt is also not a guardrail. Code that runs above the model is closer."

Layer 3: Safety as Continuous Evaluation

Agrawal emphasized that safety is not a static checkpoint but an ongoing process. While pre-launch testing is essential, the real work begins in production. This involves continuous evaluation through multiple sources:

• Automated Judges: Scalable judges scoring conversations on dimensions like clinical accuracy, safety, relevance, and drift.
• Member Feedback: Direct signals from users (e.g., thumbs up/down) that capture tone and issues missed by automated systems.
• Sampled Traces: Reviewing random samples, with 100% coverage for high-stakes cases, to catch nuanced failures.

She highlighted that the bottleneck is not compute power but the human capacity to analyze these signals. She also noted that some failures cannot be fixed solely by prompt adjustments, as models adapt. Monitoring becomes the first resort, turning new failures into new judges.

Decisioning Frameworks for Complex Scenarios

Agrawal then shifted to the second half of her talk, focusing on human decision-making when architectural safeguards are insufficient. She presented a framework for navigating disagreements among stakeholders (clinical, legal, compliance, product, engineering) when deciding whether to launch a product with known issues.

• Worst Case Wins: Severity is determined by the worst plausible outcome, not the average.
• Severity Is Not Capacity: A bug's severity is based on potential harm, not the ease of fixing it.
• Asymmetric Default: When unsure, default to the safer mistake, hold for safety bugs, ship for polish bugs.
• Revealed vs. Stated Tolerance: Calibrate to what the organization actually accepts in production, not aspirational goals.
• Humans Are the Constraint: Design with the human in the loop in mind, as judges and pattern interpretation are the true limitations, not compute power.

She concluded by summarizing the core message: "Architecture is how. Decisioning is when. Member trust is why." Building guardrails first, while slower, is essential for creating systems worthy of users' health and trust.