Nick Nisi and Zack Proser from WorkOS explore how to move "Beyond Done" with AI agents, ensuring their work is truly complete and reliable. They demonstrate building robust verification loops using custom CLIs, browser tools, and LLM judges, allowing agents to self-check and self-correct, preventing costly mistakes and accelerating development velocity for a technical audience.
Tim Sweeney, principal engineer at Weights & Biases, demos ARIA, W&B's agent for automated ML research. ARIA launches and monitors GPU training jobs on Karpathy's autoresearch project, surfaces patterns across 200+ experiments, and builds reports and dashboards. The talk then covers how the team built ARIA with Weave observability, LLM judges, YAML-based tasks as unit tests, and nightly evals, and closes with practical tips for productionizing agents.
Will Brown's talk addresses the critical challenge of applying Reinforcement Learning to real-world tasks where verifiable rewards are absent. He outlines how Primordial AI tackles this by leveraging environments as the core anchor for building reward signals. Key strategies include using LLMs as "judges," grounding tasks in production traces or document corpora, and employing "reverse direction" techniques to generate training data. Brown also details methods for calibrating task difficulty, identifying reward hacking, and fostering continual learning by treating model optimization as a science, emphasizing the use of compute to refine environmental signals and abstract human expertise.