Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Ali Khial exposes critical flaws in popular coding benchmarks, revealing how ambiguous instructions, weak verifiers, and model 'reward hacking' create a disconnect between reported performance and real-world utility. He argues that this leads to a "trust gap" where engineers disregard leaderboards. Khial then outlines five principles for building trustworthy, production-grade benchmarks, emphasizing human-authored instructions, holistic grading, economic value, contamination-free design, and informative leaderboards, urging software engineers to contribute to their improvement.