Tokenless

Evaluations

Continual Learning: How AI Agents Get Better With Every Use | Arjun Karanam, Trajectory

Continual Learning: How AI Agents Get Better With Every Use | Arjun Karanam, Trajectory

Arjun Karanam from Trajectory discusses the "experience gap" in AI, where models excel in intelligence but lack real-world experience, advocating for continual learning. He outlines four key areas for the agent ecosystem: robust traceability including corrective actions, evaluations drawn from production traffic, harnesses that orchestrate rather than constrain, and comfort with open-weight models. Trajectory aims to provide a platform for companies to own and continuously improve their AI intelligence.

Kavak's Playbook for Rebuilding a Company Around AI

Kavak's Playbook for Rebuilding a Company Around AI

Alejandro Maza Ayala, Chief Product & AI Officer at Kavak, details how the used-car marketplace transformed into an AI-native company. He explains the 'agent-per-customer' architecture, where individual agents handle 96% of customer interactions and 95% of transactions, outperforming human teams in sales (2.1x better conversion) and even acting as an 'AI CEO' that boosted profits by 50% in an experimental city. The discussion covers the need to redesign company structures, the importance of robust evaluations, and how a 'Jedi Academy' trains all employees, from executives to mechanics, to build and collaborate with AI agents. Ayala argues for 'creative destruction,' suggesting that true AI leverage comes from rebuilding organizations from the ground up, rather than incremental adoption, presenting a massive opportunity for new founders.

Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind

Philipp Schmid from Google DeepMind emphasizes the critical, often-overlooked need for rigorous evaluation of AI agent skills. He argues that shipping skills without testing is akin to deploying code without unit tests, leading to unreliable agent behavior. The talk covers what defines an agent skill, strategies for writing effective and correctly triggering skills, and a practical guide to building lightweight evaluation harnesses to catch failures proactively.

Continual System Prompt Learning for Code Agents – Aparna Dhinakaran, Arize

Continual System Prompt Learning for Code Agents – Aparna Dhinakaran, Arize

The talk by Aparna Dhinakaran introduces "system prompt learning" as an efficient alternative to traditional Reinforcement Learning for improving large language model-based coding agents. By leveraging LLM-as-a-judge evaluations to generate English feedback and explanations for code failures, agents can automatically refine their system prompts and rules. This method, demonstrated on Claude and Klein, significantly boosts performance on benchmarks like SWEBench with minimal data, highlighting the critical role of high-quality evaluation prompts.

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Building successful AI applications requires a sophisticated engineering approach that goes beyond prompt engineering. This involves creating intentionally engineered evaluations (evals) that reflect user feedback, focusing on "context engineering" to optimize tool definitions and outputs, and maintaining a flexible, model-agnostic architecture to adapt to the rapidly evolving AI landscape.

Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Practical tactics to build reliable AI apps — Dmitry Kuchin, Multinear

Moving an AI PoC from 50% to 100% reliability requires a new development paradigm. This talk introduces a practical, evaluations-first approach, reverse-engineering tests from real-world user scenarios and business outcomes to build a robust benchmark, prevent regressions, and enable confident optimization.