Diane Lin on AI Agent Inconsistency

Inconsistency in AI agent outputs is a common, yet often overlooked, challenge for developers. Diane Lin, Tech Lead for Self-Evolving AI Agents at Datadog, addressed this pervasive issue in a recent presentation, explaining its root causes and offering practical solutions. Lin, whose background spans continuous learning, one-shot learning, and zero-shot transfer learning at institutions like Imperial College London and MIT, as well as industry giants like Alexa and Vicarious (now part of Google DeepMind), highlighted that the problem often lies not with the model itself, but with the data it's trained on.

StartupHub data

Companies working on this

Profiles of the companies named in this story, with funding and a one-liner from our database.

MIT

Massachusetts Institute of Technology

Vicarious

$122M

AI company developing brain-inspired robotics solutions for complex automation tasks.

Google DeepMind

$677M

Pioneering AI research and development to solve intelligence and advance science for humanity.

Datadog

$7.8B

Cloud monitoring and observability platform for infrastructure and application performance.

Understanding the Root of Inconsistency

Lin explained that when AI agents produce inconsistent outputs for the same input, it's frequently due to data points residing in the "gray zone", the area near a decision boundary where even human experts might disagree. She illustrated this with examples from sentiment analysis and cybersecurity. For instance, a hotel review that is subtly positive but contains minor negative elements might be classified differently across multiple runs, reflecting genuine ambiguity in the data.

Similarly, in cybersecurity, a "failed login attempt from a suspicious IP" could be classified as benign or malicious depending on contextual factors or company policy. The agent's inconsistency, Lin argued, is not a flaw in the AI but rather an indication that the AI is highlighting the inherent ambiguity present in the data. Traditional machine learning models often struggle with these edge cases, leading to the observed variability in outputs.

Leveraging Active Learning and Memory for Improvement

To address this, Lin proposed leveraging active learning techniques to identify these ambiguous data points. Instead of labeling all data, active learning focuses human attention on the most informative examples, particularly those near the decision boundary. This approach can significantly improve model performance with less labeling effort.

Furthermore, Lin introduced a novel approach that augments AI agents with semantic and episodic memory. Semantic memory involves incorporating explicit domain knowledge and rules, such as company-specific policies or preferences, directly into the agent's knowledge base. Episodic memory, on the other hand, leverages past similar cases to inform current decisions. By referencing previously labeled similar instances, the agent can make more consistent judgments without requiring explicit rule-based knowledge distillation.

The combination of these two memory systems, Lin explained, allows for a more robust and adaptable AI agent. Episodic memory handles recurring situations automatically, freeing up human reviewers to focus on truly ambiguous cases that require deeper analysis and the addition of new domain knowledge to the semantic memory.

Tangible Benefits and Experimental Results

Lin presented experimental results demonstrating the effectiveness of this approach. By applying their solution, which incorporates both semantic and episodic memory, they were able to significantly improve the consistency of AI agent verdicts. In an experiment with 93 cybersecurity alerts, a quarter of which initially showed inconsistent verdicts across three reruns, their method reduced the remaining inconsistency. This leads to more trustworthy AI agents, more efficient quality control, and better alignment with customer-specific needs and preferences.

Lin concluded with three key takeaways for AI developers:

• Inconsistency is often a data problem, not a model problem. Focus on label ambiguity and insufficient information.
• Model disagreement is a feature, not a bug, signaling areas for learning.
• Fine-tuning is not the only path to improvement; augmenting agents with semantic and episodic memory offers a more efficient alternative.

By embracing these principles, developers can build more reliable and adaptable AI agents that better serve user needs.