Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog

This session, led by Dr. Diane Lin from Datadog, addresses the critical issue of AI agents, particularly Large Language Models (LLMs), exhibiting inconsistent behavior—providing semantically different outputs for the same input across multiple runs. While often dismissed as the “stochastic nature of LLMs,” this inconsistency creates significant trust issues for users, especially in high-stakes fields like cybersecurity where a “flip-flop” between a malicious threat and a benign alert can have disastrous consequences.

Dr. Lin argues that these inconsistencies are typically not model failures. Instead, they concentrate in what’s termed the “gray zone” around the decision boundary. These are cases where policies are ambiguous, input information is insufficient, or even human experts might disagree. Examples include:

• Sentiment Analysis: A hotel review containing both positive and slightly negative aspects, leading to inconsistent positive/negative labels by an LLM, reflecting human disagreement.
• Cybersecurity: Alerts indicating failed login attempts from suspicious IPs. Depending on subtle factors (e.g., whether the attacker gained access, company policy on early warnings), the same alert could be flagged as benign or malicious, again highlighting policy ambiguity.

The core message is that the AI agent isn’t faulty; it’s revealing inherent ambiguities that require clarification.

Solution: Active Learning combined with Memory Augmentation

The proposed solution leverages well-established machine learning concepts like Active Learning to identify these problematic “gray zone” data points, but with crucial adaptations for the LLM era, coupled with Semantic and Episodic Memory augmentation rather than expensive fine-tuning.

1. Finding the Gray Zone with Active Learning:

   • Traditional Active Learning identifies data points where the model is uncertain (e.g., probability close to 0.5) or where a committee of models disagrees.
   • For LLMs, Dr. Lin notes that internal uncertainty scores are often unreliable (“LM model doesn’t know what it doesn’t know”).
   • The recommended approach is to identify disagreement across multiple runs of the same model or predictions from different models. These “flip-flopping” cases reliably signal areas needing human guidance.
   • Once identified, these cases are presented to human experts for review, clarification, and labeling.

2. Augmenting Agents with Semantic Memory:

   • Instead of immediate model retraining, the first step after human review is to distill the clarified policies and domain knowledge into the agent’s semantic memory.
   • This involves explicitly defining rules or guidelines that resolve the ambiguity. For instance:
     • For hotel reviews: “If a customer complains about something outside the hotel’s control, classify as positive to avoid unnecessary attention.”
     • For cybersecurity: “If a password spray attempt does not result in a successful login, classify as benign; if it succeeds, classify as malicious.”
   • This process sharpens decision boundaries and improves consistency for both the AI agent and human experts.

3. Augmenting Agents with Episodic Memory:

   • Episodic memory involves storing past similar cases along with their human-reviewed decisions and referencing them automatically.
   • This approach is particularly effective for recurring situations, such as common false positives in cybersecurity, allowing the agent to automate decisions without requiring immediate distillation into semantic rules.
   • It provides a “low-hanging fruit” method for immediate consistency improvement, especially where formal policy distillation might take time.

Complementary Approach:

Semantic and episodic memories are complementary. Episodic memory handles recurring cases automatically. For cases that episodic memory cannot resolve (new cases or those without clear past references), they are passed to human reviewers. These human reviews then contribute to refining semantic memory, creating a continuous learning loop.

Benefits and Results:

This methodology yields several key benefits:

• Improved Consistency and Trust: Significantly reduces “flip-flops,” making AI agents more reliable.
• Efficient Quality Control: Human review is focused only on genuinely ambiguous cases, maximizing the return on labeling effort.
• Customer Adaptation: The process continuously gathers customer-specific preferences and policies, allowing the agent to evolve and adapt to individual customer environments.

Experimental results from 93 cybersecurity alerts showed that without this solution, a quarter of alerts “flip-flopped.” With episodic memory, 15% became consistent. The remaining 10% inconsistent cases were then addressed by human review, leading to further consistency and adaptation.

Key Takeaways:

• Inconsistency is often a data/policy problem, not a model failure.
• Model disagreement is a feature, not a bug, providing a signal for learning.
• Augmenting agents with semantic and episodic memory is a powerful, lighter-weight alternative to expensive fine-tuning, fostering continuous improvement and adaptation.

This approach transforms inconsistency from a product flaw into a powerful mechanism for continuous agent improvement and alignment with specific customer needs.

• Tags:
• AI Agents
• Data Quality
• Trustworthy AI
• Continuous Learning
• Active Learning
• Episodic Memory
• Cybersecurity AI
• LLM Inconsistency
• Stochasticity
• Decision Boundary
• Gray Zone
• Semantic Memory
• Model Consistency
• Policy Clarification