Oct 05, 2026

From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

This presentation by Jeff Koss and Leo Platzer from Deasy Labs (now Collibra) tackles the critical challenge of scaling AI chatbots from pilot projects to production with large, messy unstructured datasets. They demonstrate a solution for curating and cleaning 80,000+ SharePoint documents, highlighting the importance of data quality—specifically identifying duplicates, conflicts, sensitive information, and ensuring data freshness—to significantly improve RAG chatbot performance and reduce development time.

Oct 05, 2026

The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize

Fuad Ali from Arize AI explains why voice agents are uniquely challenging to debug, as critical issues like latency, interruptions, and tone are invisible in text logs. He presents a solution combining unified audio, transcript, and trace views with OpenInference semantic conventions and audio-native evaluations. The talk culminates in a vision for self-healing voice agents through automated observation, evaluation, and improvement cycles, demonstrating how to achieve robust, continuously improving AI voice systems.

Oct 05, 2026

Evals in AI: A Deep Dive — Tejas Kumar, IBM

This deep dive into AI evaluation (evals) starts by clarifying their role in providing ahead-of-time reliability and security for non-deterministic systems, contrasted with just-in-time harnesses. Tejas Kumar defines evals as "fuzzy unit tests" and outlines their core components: test cases, expected behavior, a judge, and aggregation. He explores various evaluation techniques, emphasizing the challenges and biases of using LLMs as judges, such as position bias, sycophancy, self-preference, and verbosity. The presentation details a practical methodology for calibrating LLM judges against human verdicts using a "golden dataset" to achieve an 80-85% agreement threshold, which can then be enforced as a CI/CD gate. A live coding demonstration illustrates the evolution from simple string matching to a policy-aware LLM judge, showcasing how refining policy and incorporating RAG (Retrieval Augmented Generation) via OpenRAG dramatically improves decision-making. The discussion also covers strategies for maintaining living datasets in production and managing evaluation costs.

Oct 05, 2026

Your AI Agent Is Confidently Wrong About Production — Willem Pienaar, Cleric

Willem Pienaar, CTO of Cleric, details why AI agents struggle with production debugging compared to code generation. He explains the unique challenges of dynamic, unverified production environments and LLMs' inherent confidence bias. Pienaar outlines four common agent failure modes (symptom/cause confusion, missing baselines, abstraction loss, anchoring) and introduces grounding techniques—like generating multiple theories, using baselines and dependency graphs, temporal analysis, and evidence propagation—to enhance diagnostic accuracy. He also emphasizes the critical need for verifying fixes against real outcomes to calibrate agent confidence, ultimately aiming to bring production signals into the development environment for proactive failure prediction.

Oct 05, 2026

Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

This talk by Jess Wang from Braintrust delves into the critical importance of evaluations (evals) in AI development, moving beyond subjective "vibes" to data-driven decision-making. She breaks down the four core components of an eval—dataset, task, scoring, and experiments—and illustrates the continuous improvement "eval flywheel" integrating observability. The central experiment compares agentic search against vector search for a coding agent tasked with finding buggy code in Microsoft's TypeScript Go repo. While both methods achieved identical accuracy, vector search proved four times more expensive due to its inability to capture surrounding context within code chunks, leading to excessive, inefficient searches. Wang concludes with practical caveats and best practices for conducting effective LLM evals.

Oct 05, 2026

How a Logistics Giant Keeps AI Data Locked Down

Jason Ward, FinOps for AI lead at C.H. Robinson, shares insights on managing AI costs for enterprises and startups. He details a FinOps strategy centered on comprehensive tagging, leveraging AI for observability (e.g., detecting prompt bloat and context starvation), and pioneering advanced metrics like "cost per thought" and "reasoning ratio" to optimize complex AI workloads beyond simple discrete tasks.