1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

Vishal Srivastava and Tanmay Sah of New York Life analyze 1,368 public AI incidents to show what real failures reveal about how AI systems should be tested. They organize failures into three dimensions (technical strength, deployment controls, business impact) and 13 sub-dimensions covering about 98% of cases, find that weak deployment controls are most closely tied to harm, and present a materiality-based ranking of use cases, a failure-driven build-break-fix evaluation loop, and an airline chatbot refund case as an example.