1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

AI failures are rising fast, and enterprises now deploy far more AI use cases than teams can review one by one. Vishal Srivastava and Tanmay Sah argue that a structured way to learn from failures is needed, and they built one from 1,368 incidents in a public AI incident database.

Three dimensions of failure

The framework separates failures into three core dimensions. Technical strength covers the development stage, where data quality, architecture, and system design can break. Deployment controls cover what happens once the system is live: whether errors are contained before they reach production and customers. Business context covers the consequences, including reputational, legal, and financial risk, which depend on where the system is used. The central point is that the model alone is only one part of the system, and it has to be evaluated holistically.

Thirteen sub-dimensions

Technical strength breaks into six sub-dimensions: data quality, system design for the intended task, predictive validity and outcome analysis, fairness, failure modes (where, when, and for whom the system breaks), and explainability for regulators and other stakeholders. Deployment controls break into four: decision authority (whether the AI informs or acts on someone’s behalf), human mediation (how effective human-in-the-loop review is), exposure control (internal users who can catch errors versus external customers where small issues spread fast), and monitoring. Business impact has three: financial impact, legal consequences, and scale, meaning how many users are affected. Together these thirteen sub-dimensions cover roughly 98% of the incidents studied, and a typical incident involves about six of them at once.

Which failures cause the most harm

Not all sub-dimensions carry equal risk. Plotting harm association against frequency shows that weak human mediation and missing harm detection are most tightly tied to harm, while scale and system design appear often but are less associated with harm. Grouping the sub-dimensions, deployment controls are the most closely linked to harm, followed by business impact, then technical failure. The practical implication is that once deployment controls are in place for a high-risk system, a meaningful level of assurance follows, and resources should be allocated by harm potential rather than spread evenly across all failure types.

Ranking use cases by materiality

This leads to a materiality scale for ranking AI use cases by how much is at stake. Because materiality tracks harm directly, teams can concentrate evaluation effort on the most material use cases instead of treating every use case the same.

Evaluating the whole deployment system

Tanmay Sah extends the logic into system-level evaluation by decomposing risk into four components: how often the model fails, whether the failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. The same model can carry different risk depending on how it is deployed, including how much authority it holds, whether anyone reviews its output, and whether monitoring can catch errors before they cause harm. The evaluation therefore has to cover the deployment setup as well as the model.

Traditional ML testing checks held-out data, metrics such as accuracy and AUC-ROC, and the assumptions behind them. Agentic systems add many more surfaces: retrieval quality, tool failures, the sequence of actions taken across a trajectory, whether human review catches mistakes, and whether the agent stays within its permissions and can recover. Scenarios are often generated or judged by another AI agent or judge, and that judge can make mistakes and miss important failures. For this reason the evaluator itself is part of the system that must be governed, with its reliability, the coverage of its evidence, and its applicability to the deployed system all checked.

Confirmation bias and failure-driven evals

Vishal Srivastava draws on cognitive science to make the case for falsification. Peter Wason’s 2-4-6 experiment showed that people tend to test hypotheses by looking for confirming examples, such as proposing even numbers, when the rule was simply any increasing sequence. In that study, 79% of participants got the rule wrong. Since AI systems are trained on human-generated data, they can inherit the same bias. The proposal is that evaluations should be failure-driven: start from the system’s claims, try to break them, and identify exactly where they fail.

Build, break, fix

The resulting method is an iterative build-break-fix loop. Each cycle maps the region where the system can be trusted, the uncertain region, and the high-risk region where its performance cannot be assumed. Where failures appear, teams either restrict the system from operating in that territory or add supporting controls they can be confident in, then repeat until the level of confidence is adequate. Reliability and materiality together determine where the AI is allowed to operate, so a high-risk use case stays inside the region that has been verified.

A real chatbot failure

To make this concrete, Tanmay Sah walks through a case in which a customer travelling after a family death asked an airline chatbot about a discounted fare. The chatbot advised buying a regular ticket and claiming the discount within 90 days. The airline later refused, since its policy did not allow that request, and a tribunal held the airline responsible. The failure chain runs from incorrect advice, to the customer acting on it, to the organization absorbing the consequences.

The build-break-fix approach would have exposed this earlier. The starting requirement is that advice must match the approved policy. Challenging the chatbot with questions about refunds after travel would surface the incorrect advice. A policy check or human escalation could then be added, and the system retested to confirm those controls catch errors before they reach a customer. The method is to define the claim, challenge it actively, use the evidence to guide changes, and then make the deployment decision.

Takeaway

Evaluations should be driven by risk rather than frequency. Break the entire system into core components (technical failure, deployment controls, and business context), assess each use case’s materiality on a scale that can be tailored to the company and updated as risk changes, and apply system-level stress testing. Deployment decisions should follow only once confidence in each region has been established, because otherwise the cost of failure may be too high. The core message is to evaluate the full AI system based on risk, not just one component.

• Tags:
• Risk Assessment
• AI Governance
• Feature
• LLM Evaluation
• Agentic Systems
• AI Evaluation
• Model Monitoring
• Human-in-the-Loop
• AI Failures
• AI Incident Database
• Deployment Controls
• Materiality
• Confirmation Bias
• Failure-Driven Evals
• Build Break Fix
• AIE CODE