The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI

Rayan Krishnan of Vals AI explains why single benchmark scores in model cards hide a jagged frontier, using finance tasks like tax and Excel LBO modeling to show that the best model changes by task and that cheaper models can cost more. He covers benchmark design lessons (accuracy metrics, validity, reliability, reward hacking, multi-turn work), a case study in which a team cut token spend about 90% using internal evals built from its own GitHub history, and why a firm's own work history is its most valuable eval source.