Oct 22, 2026

1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

Vishal Srivastava and Tanmay Sah of New York Life analyze 1,368 public AI incidents to show what real failures reveal about how AI systems should be tested. They organize failures into three dimensions (technical strength, deployment controls, business impact) and 13 sub-dimensions covering about 98% of cases, find that weak deployment controls are most closely tied to harm, and present a materiality-based ranking of use cases, a failure-driven build-break-fix evaluation loop, and an airline chatbot refund case as an example.

Oct 06, 2026

What AI gets wrong and what failure teaches us

Jennifer Neville discusses her research into "surprising failures" in AI systems, where models, particularly LLMs, struggle with complexity in realistic user workflows. Her team focuses on practical evaluation for multi-turn conversations and agentic knowledge work, emphasizing the need for robust AI that understands context and user intent over time. She offers insights into bridging the gap between user expectations and AI capabilities, advocating for detailed user feedback and outlining the future of human-AI collaboration.

Oct 06, 2026

Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

Doug Guthrie showcases how Braintrust connects observability with AI agent improvement. Using a support agent, he demonstrates setting up tracing, establishing evaluation baselines, and publishing custom scorers. The workshop delves into Braintrust's Topics feature for discovering unknown failure modes, configuring online automations, and importing real-world traces. A key focus is on automating the improvement cycle: using the Loop assistant for SQL-based trace analysis, integrating coding agents with Braintrust CLI skills to propose code changes, add regressions, run evals, and generate pull requests, thereby closing the feedback loop from production to development.

Oct 01, 2026

Better AI Starts With Better Verification

Halluminate, a Y Combinator S25 company, builds RL environments and benchmarks that teach AI models to do non-coding knowledge work, starting with financial services. In this fireside with YC Partner Jon Xu, co-founders Jerry Wu and Wyatt Marshall trace their pivot from evals to training environments, explain why verification quality matters more than task volume, and outline a roadmap toward multi-agent simulated companies. They also address reward hacking, alignment as an industry-wide responsibility, and what they look for in new hires.

Sep 09, 2026

Inside the Race to Measure Frontier Intelligence

Independent model evaluation is crucial for AI progress, as public benchmarks are saturated and models optimize for them. Rayan Krishnan discusses VALS's role in providing continuously evolving, third-party evaluations, highlighting the Llama 4 'disaster' and the need to measure increasingly agentic systems. The conversation covers the complexities of evaluation beyond capability to include cost and latency, the looming financial impact of token spend on enterprises, and the critical role of evaluations in informing AI policy and international coordination for aligning AI with human intent and managing geopolitical risks, especially concerning recursive self-improvement.

Aug 30, 2026

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

This detailed discussion with Google AI researchers highlights the latest generative video models (NanoBanana 2 Lite, Gemini Omni Flash APIs) and their applications, including video editing, education, and localization. The conversation critically examines whether language is an adequate intermediate representation for AI, especially for sensory experiences like audio and taste, where human vocabulary is limited. It exposes the unreliability of human preferences in evaluating AI-generated content, citing the "Instagram filter effect" and "wedding ring" reward hacking as examples. The researchers emphasize the importance of high-quality, real-world data and the critical role of forward-deployed engineers in bridging the gap between cutting-edge research and practical applications.