Jul 31, 2026

Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Ali Khial exposes critical flaws in popular coding benchmarks, revealing how ambiguous instructions, weak verifiers, and model 'reward hacking' create a disconnect between reported performance and real-world utility. He argues that this leads to a "trust gap" where engineers disregard leaderboards. Khial then outlines five principles for building trustworthy, production-grade benchmarks, emphasizing human-authored instructions, holistic grading, economic value, contamination-free design, and informative leaderboards, urging software engineers to contribute to their improvement.

Jul 12, 2026

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

This talk introduces the application of psychometrics, particularly Item Response Theory (IRT), to improve LLM evaluation. It highlights how IRT goes beyond simple accuracy to measure model ability, item difficulty, and discrimination, enabling benchmark auditing, adaptive testing, data leakage detection, and the identification of model relationships and distillation, ultimately providing deeper insights into what LLMs truly learn.