Catastrophic agent failure and how to avoid it // Edward Upton // Agents in Production 2025
Edward, a founding engineer at Asteroid, discusses the critical challenge of managing catastrophic failures in agentic browser solutions, particularly in high-stakes domains like healthcare and insurance. He shares real-world examples of agent failures and outlines a practical framework for building more reliable, predictable, and accountable agents by scoping their capabilities, implementing robust human-in-the-loop tooling, and employing independent evaluation systems.
In the realm of agentic browser automation, customers expect agents to replace human tasks with the same quality, demanding human-level accuracy, consistency, and accountability. This is especially true in sensitive domains like healthcare and insurance, where failures can have severe real-world consequences. The core lesson is that a truly “working” agent is one that achieves zero false positives. A false negative (failing to complete a task it could have) is acceptable because it can be manually rectified without negative side effects. A false positive (incorrectly reporting success) can lead to irreversible, catastrophic outcomes.
What Makes a Failure Catastrophic?
A failure becomes catastrophic when two conditions are met:
- Real-World Side Effects: The agent is not operating in a sandbox but has capabilities to interact with and change the state of the real world, such as accessing terminals or modifying data in web applications.
- Irreversibility: The agent’s actions cannot be simply undone. Browser automation inherently involves state changes—clicking buttons, submitting forms, navigating pages—that build upon each other. Once a wrong action is taken, reverting to a previous state is often impossible.
When a failure occurs under these conditions, it’s crucial to either prevent it entirely or have systems to immediately identify it for human intervention.
Case Study 1: The Scam Intelligence Agent
A product designed to mine scam intelligence from emails provides a stark example of failure.
- The Scenario: A bank accidentally sent an overdraft notice to a mistyped email address, which happened to be a spam trap monitored by the system. The agent flagged this as a potential scam email.
- The Failure: The system included a human approval step. However, the tooling was poor, and the human reviewer, faced with thousands of tedious approvals, mistakenly authorized the agent to proceed.
- The Consequence: The agent began interacting with the bank, impersonating the account holder. The bank grew suspicious and, believing they were dealing with a fraudulent human actor, sent a notification that they were blocking the real user’s account.
- The Learning: Human-in-the-loop is essential, but it is only as effective as the tooling provided. If the process is tedious and error-prone, it offers no more safety than having a human perform the entire task.
Case Study 2: The Healthcare Appointment Agent
A second example from Asteroid’s work highlights the danger of flawed success evaluation.
- The Scenario: An agent was tasked with booking a healthcare appointment for a user based on provided symptoms.
- The Failure: The agent started filling out the form but took an incorrect branch. Based on the path it took, it concluded an appointment wasn’t necessary and did not book one. However, it still reached a generic confirmation page (e.g., “Thank you for your inquiry”). The agent interpreted this page as a successful completion of its task.
- The Consequence: The system reported a “success,” and no one was aware that the user’s critical appointment was never booked. This was a catastrophic failure with potentially serious health implications, made invisible by an incorrect evaluation of the outcome.
- The Learning: An agent’s own assessment of its success cannot be trusted. A separate, robust evaluation system is needed to verify that the final state matches the initial goal.
A Framework for Building Reliable Agents
To combat these failures, Asteroid has developed a three-part strategy focused on predictability, visibility, and rigorous evaluation.
1. Scope the Agent’s Capabilities
Instead of building fully autonomous, ReAct-style agents that are given a high-level prompt and left to their own devices, the focus is on creating more predictable, structured systems.
- Predictability over Autonomy: Customers in high-stakes environments are willing to trade some development time for more predictable and reliable agent behavior.
- Embrace Failure: Agents are designed without a bias to “keep going.” They can and should fail out if they encounter uncertainty. This prioritizes avoiding false positives. A false negative is always preferable to a false positive.
2. Provide Excellent Tooling and Visibility
Effective human-in-the-loop processes are critical for building, debugging, and maintaining trust in agents.
- Collaborative Building: Agent development is an iterative process. A platform with clear visibility—including recordings and step-by-step logs—allows technical users to debug and gives stakeholders confidence that the agent is following a defined path.
- Intelligent Escalation: Instead of just failing, the agent should be able to escalate to a human when it gets stuck (e.g., due to missing input data). This is vital if the agent has already performed partial actions, as it allows for manual recovery and provides feedback to improve the agent.
3. Implement Robust, Independent Evaluation
An agent’s reported outcome should never be the final word on its success.
- Decoupled Evaluation: A discrete evaluation system should run after the agent completes its execution. This system’s sole purpose is to critically analyze the entire execution log and determine if the outcome truly matches the initial objective.
- Flexible Evaluators: Success criteria vary. For one customer, reaching the end of a flow is a success. For a QA testing agent, reaching a page that should have been inaccessible is a failure. The evaluation system must be flexible enough to accommodate custom, domain-specific rules. For example, a rule could be: “The task is only successful if the final page contains the exact text ‘Your appointment is confirmed for…’”. This approach drastically reduces the false positive rate.
Ultimately, the trade-off between pure autonomy and deterministic safety is central. By using a graph-based platform, developers can mix deterministic nodes (e.g., a script for logging in) with more flexible, agentic nodes, finding the right balance to build production-ready agents that are both powerful and trustworthy.