AI Engineer hosted Upstart voice AI lead Shiv Indap as he showed how a single follow-up question can turn a lending verification call into a compliance violation.
Upstart, a fintech lender, auto-decisions about 90% of applicants, Indap said. The other 10% must verify three things by phone: income continuity for three months, planned use of funds, and outstanding obligations like alimony. The trap is remote and conversational. If a borrower says medical use, the agent cannot ask the amount. If they say travel for a graduation, the agent must clarify whose graduation, since the answer changes the purpose. Applicant curiosity is a third trap, as borrowers push for status updates that the agent must politely deflect.
The gap was tribal knowledge.
Early review ran on a Google Sheet that operations used to audit every call. Five columns became ten, then fifteen, with random edits, Indap said. The team replaced it with structured human review built with Brooke Hopkins, founder of Coval, whose background is evaluation infrastructure at Waymo. They defined LLM as judge as an automated scorer that runs after each call and grades compliance yes or no with an explanation. They tracked accuracy, clarity, completeness, policy alignment, escalation and CSAT, then winnowed to what mattered. Human reviewers checked the judge output, and agreement on compliance started at 33%. Encoding unwritten rules and aligning reviewer guidance lifted it to 92%, and changes to the agent or to the judge were gated behind feature flags so engineering or compliance could sign off.
Hopkins framed evals as product definition and sorted them into three types: did the agent reach the end, did it follow steps in order, and how was the conversation quality. Her loop monitors live calls for known failures, like missed verification wording, and finds unknown failures with clustering and anomaly detection, then routes them. Compliance violations go to compliance for action, script breaks go to the agent team. Real failures are re-simulated with voice-to-voice testing that exercises accents, background noise and interruptions, so fixes are proven before they touch users. No system hits 100% at scale.
The daily brief
Get the free daily brief with the day's AI news, new AI startups and funding rounds every morning.
Join 12,005 subscribers.
That discipline is spreading because the base tech is now cheap to try. Voice AI lets builders turn written content into natural-sounding speech in seconds and even clone a brand voice, which startups already use to prototype faster. For lending and other regulated support, that ease raises the bar for deployment. Teams must budget for a permanent human review queue and simulation harness, not just prompt work, and they must track judge-human agreement as a release metric.
Indap and Hopkins said self-healing does not mean humans leave the loop. Watch whether your routing sends each failure to the right owner and whether simulation reproduces it before you ship. That is the workflow that has to change now.
