Duolingo's Angel Lee on AI Discernment vs. Approval

Angel Ortmann Lee, a software engineer at Duolingo, delivered a compelling presentation titled "Build AI Systems for Discernment, Not Approval." Lee's talk emphasized a critical shift in how we design and interact with AI systems, moving beyond simple approval to fostering genuine discernment. The core message revolved around the importance of engineering human-AI interactions that promote critical thinking and accountability, especially in high-stakes applications.

Understanding Human-in-the-Loop AI

Lee began by defining human-in-the-loop AI as a system where a human actively participates in the operation, supervision, or decision-making of an automated system. This human involvement is typically intended to ensure accuracy, safety, accountability, or ethical decision-making. The process is often visualized as a linear flow: Model → Human → Decision. However, Lee highlighted that this process is not strictly linear but cyclical, with feedback loops influencing subsequent stages.

The Rise of Automation Bias

The presentation touched upon the growing trust in technology and how it affects human behavior. As AI becomes more integrated into daily life, trust tends to increase while caution decreases. This phenomenon, known as automation bias, was illustrated with examples like using GPS navigation without critical thought or relying on search engine results without verification. Lee cited a study where participants accepted AI-generated answers on an exam 80% of the time, even when the answers were wrong, demonstrating a significant reliance on AI output.

This reliance can lead to humans either supplementing or supplanting their own thinking with AI, often without realizing it. This is particularly concerning in contexts like college admissions or visa applications, where errors can have significant consequences. The research indicated that when AI was correct, human performance improved by 25%, but when the AI was wrong, human performance decreased by 15%, highlighting the detrimental effects of automation bias.

Case Study: Duolingo English Test

Lee then delved into a case study on the Duolingo English Test (DET), a high-stakes online English proficiency exam accepted by over 6,000 institutions worldwide. To ensure score legitimacy and integrity, the DET employs multiple layers of security, including identity verification, a locked-down testing environment, AI-assisted monitoring, and human proctor review. A specific focus was placed on AI-assisted monitoring for "copy-typing," which involves reproducing text from another source instead of composing it independently.

Duolingo uses a CNN-Transformer model that analyzes keystroke patterns to distinguish transcription from composition. When copy-typing is flagged, human proctors verify the situation. The model employs a conservative threshold, aiming for a low false positive rate. However, an experiment was conducted to test human reviewers' susceptibility to automation bias. By introducing fake AI signals for copy-typing, the study found that reviewers endorsed these false alerts at near coin-flip rates, indicating evidence of automation bias.

Addressing Automation Bias Through Design

To combat these signs of automation bias, Lee explained that Duolingo targeted the human-AI interaction loop. Updated proctoring guidelines were implemented to emphasize that an AI signal should be treated as a preliminary alert, requiring human reviewers to find independent evidence before upholding a flag. This shift from a purely AI-driven flagging system to one that requires human discernment was crucial.

The key principle here is to "engineer the reasoning." AI developers should consider what reasoning patterns are needed from humans and how the interface can elicit them. This involves:

• Structuring inputs and outputs to be specific, not just walls of text.
• Highlighting assumptions made by the model and asking for sign-off.
• Building in friction and review gates to slow down deliberate thought where needed.
• Collecting explicit feedback to create a taxonomy of signals for model evaluation and improvement.

The Virtuous Cycle of AI Interaction

The choice of interaction design determines whether the system operates in a vicious cycle (model makes confident calls, humans rubber-stamp, AI becomes more confident) or a virtuous cycle (interface forces independent judgment, disagreements are logged, model improves where it's wrong). Lee stressed that every interaction is already a label, providing data that can be used for analytics, model improvements, and future innovation.

Ultimately, the goal is to design AI systems that foster discernment, enabling humans to act as investigators rather than mere validators. This requires a deliberate approach to interaction design, ensuring that the system provides clear, addressable, and actionable feedback that mimics human review patterns, ultimately leading to more accurate and reliable AI outcomes.