Oct 05, 2026

Evals in AI: A Deep Dive — Tejas Kumar, IBM

This deep dive into AI evaluation (evals) starts by clarifying their role in providing ahead-of-time reliability and security for non-deterministic systems, contrasted with just-in-time harnesses. Tejas Kumar defines evals as "fuzzy unit tests" and outlines their core components: test cases, expected behavior, a judge, and aggregation. He explores various evaluation techniques, emphasizing the challenges and biases of using LLMs as judges, such as position bias, sycophancy, self-preference, and verbosity. The presentation details a practical methodology for calibrating LLM judges against human verdicts using a "golden dataset" to achieve an 80-85% agreement threshold, which can then be enforced as a CI/CD gate. A live coding demonstration illustrates the evolution from simple string matching to a policy-aware LLM judge, showcasing how refining policy and incorporating RAG (Retrieval Augmented Generation) via OpenRAG dramatically improves decision-making. The discussion also covers strategies for maintaining living datasets in production and managing evaluation costs.

Jul 31, 2026

Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Ali Khial exposes critical flaws in popular coding benchmarks, revealing how ambiguous instructions, weak verifiers, and model 'reward hacking' create a disconnect between reported performance and real-world utility. He argues that this leads to a "trust gap" where engineers disregard leaderboards. Khial then outlines five principles for building trustworthy, production-grade benchmarks, emphasizing human-authored instructions, holistic grading, economic value, contamination-free design, and informative leaderboards, urging software engineers to contribute to their improvement.

Jul 12, 2026

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

This talk introduces the application of psychometrics, particularly Item Response Theory (IRT), to improve LLM evaluation. It highlights how IRT goes beyond simple accuracy to measure model ability, item difficulty, and discrimination, enabling benchmark auditing, adaptive testing, data leakage detection, and the identification of model relationships and distillation, ultimately providing deeper insights into what LLMs truly learn.