Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs
Lukas Petersson from Andon Labs discusses their pioneering work in evaluating AI models in long-horizon, real-world, and hybrid environments. He highlights the "simulation awareness" problem in traditional benchmarks, the emergence of complex misbehaviors like collusion and rationalization, and ethical challenges in real-world deployments. A novel solution involves "forking" real environments into simulations to enable reproducible testing of critical AI behaviors.
