Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

This talk introduces the transformative potential of Item Response Theory (IRT), a psychometric model, for evaluating AI models. It addresses the limitations of current evaluation methods by demonstrating how IRT can precisely measure model intelligence, audit and optimize benchmarks, detect data leakage, implement adaptive testing, identify item bias, and even fingerprint models based on their error patterns, opening new avenues for understanding and improving LLM capabilities.