Nick Nisi and Zack Proser from WorkOS explore how to move "Beyond Done" with AI agents, ensuring their work is truly complete and reliable. They demonstrate building robust verification loops using custom CLIs, browser tools, and LLM judges, allowing agents to self-check and self-correct, preventing costly mistakes and accelerating development velocity for a technical audience.
A talk on why coding agents shift the bottleneck from code generation to CI, merge queues, review, and approvals, and how to redesign those systems around speed, automated repair, policy enforcement, and auditable records.
A Q&A from the final part of Anthropic's Claude Managed Agents workshop, covering CI/CD for agent configs in YAML with the ant CLI, eval practices, workspace-based resource sharing and tenant isolation, agent-driven code review, GPU-enabled self-hosted sandboxes, multiplayer Slack agents, long-running sessions, and the coordinator pattern for custom tools.
Ali-Reza Adl-Tabatabai, co-founder of Gitar (now part of Sonar), explains why AI-driven growth in PR volume and size turns CI and code review into a bottleneck, and presents agentic validation as the way out: automated review, CI failure analysis, autofix, and auto-merge, built up step by step as trust grows, combined with traditional program analysis in SonarQube.
This deep dive into AI evaluation (evals) starts by clarifying their role in providing ahead-of-time reliability and security for non-deterministic systems, contrasted with just-in-time harnesses. Tejas Kumar defines evals as "fuzzy unit tests" and outlines their core components: test cases, expected behavior, a judge, and aggregation. He explores various evaluation techniques, emphasizing the challenges and biases of using LLMs as judges, such as position bias, sycophancy, self-preference, and verbosity. The presentation details a practical methodology for calibrating LLM judges against human verdicts using a "golden dataset" to achieve an 80-85% agreement threshold, which can then be enforced as a CI/CD gate. A live coding demonstration illustrates the evolution from simple string matching to a policy-aware LLM judge, showcasing how refining policy and incorporating RAG (Retrieval Augmented Generation) via OpenRAG dramatically improves decision-making. The discussion also covers strategies for maintaining living datasets in production and managing evaluation costs.
Raindrop is building the safety layer for AI agents, detecting and preventing production failures like hallucinations and failed tool calls. The company addresses the paradox that as agents become more capable, they introduce greater complexity and more ways for things to go wrong. Their new Raindrop Simulations product catches critical failures in CI before they ever reach production, democratizing a capability previously limited to large AI labs.