Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

This workshop, “Beyond Done,” by Nick Nisi and Zack Proser from WorkOS addresses the critical challenge of trusting AI agents, especially when they claim tasks are complete. They open with a stark example: an agent deleting production databases in 9 seconds, highlighting the need for robust verification.

The Problem: “Done” Isn’t Always Done

Traditional code review and manual checks are insufficient for agents, which generate code much faster than humans. Reviewing every line is unsustainable and leads to a high signal-to-noise ratio. The core issue is the gap between an agent’s self-reported completion and actual correctness, leading to subtle bugs.

Building Trust: Verifiable Goals and Feedback Loops

The central thesis is that if an agent can verify its own work, nothing else matters (Lauren Tan). Verification is foundational for achieving true scale with agents and enabling recursive self-improvement (RSI) in codebases.

Giving Agents Goals They Can Check

Agents need clear, defined, and observable goals. Vague instructions lead to assumptions that may not align with human expectations. Instead of relying on agent self-reporting, explicit verification mechanisms are crucial. This shifts the focus from code quality to the result and value delivered.

Tools for Verification

To enable agents to verify their work, they need tools that mimic human developer checks:

• Browser tools (e.g., Playwright): For checking UI interactions, form submissions, success messages, etc.
• Custom CLIs: Nick’s Ryker and Zack’s CTX are examples. CTX centralizes verification authority, checking things like PR health and merge checks via GitHub API. Ryker focuses on babysitting PRs through reviews and code fixes, ensuring overall goals are met.
• The snack CLI: A workshop-specific tool demonstrating how to create custom verification. It includes commands for:
  • Health checks: doctor (Node, Chromium, session, build status).
  • Interaction: join (adding to a waitlist), reload, reset.
  • Observation: snapshot (JSON representation of page state), screenshot, network, console logs, local storage.
  • Proof: prove (deterministic checks against expected output), judge (using System One models for flexible, non-exact matches).

Making Everything Feedback-Loopable

The goal is to create durable, reliable, and repeatable feedback loops. If an agent fails a verification, that failure should drive the next fix instead of landing on a human’s desk. This allows agents to self-heal and self-correct, dramatically increasing velocity.

• Good error messages: Explicit and detailed error messages enable agents to understand why something failed and self-heal more effectively.
• Outcome-driven development: Focus on the desired outcome, not overly prescriptive steps. Frontier models (e.g., Fable, Astra, Opus) are increasingly capable of creative problem-solving within constraints.
• Red, Green, Refactor: The familiar TDD loop applies to agents: verify (red), fix (green), improve (refactor).

LLM Judges and Eval Baselines

Static, one-to-one text comparisons for verification can be brittle. This is where System One models (fast, intuitive decisions) come in as “judges.” Instead of an exact string match, an LLM judge can evaluate if the spirit of the desired outcome is met (e.g., “you’re in, your seat is confirmed” instead of “waitlisted”).

• Saboteur and Detective game: A demo showing how a random bug can be introduced, and an agent (or human) uses verification tools to find and prove its existence.
• Eval baselines: Establishing clear criteria for success that evolve with the team. Consistent use of evals allows for measuring improvements or regressions.

Keeping Receipts and Future Implications

Every agent run produces valuable data (“receipts”). These records of past performance are crucial for:

• Historical data: Feeding into LLMs for current fixes.
• Future RSI: As tooling improves, these data sets will be invaluable for training self-evolving codebases and hill-climbing agents.
• Automated oversight: Agents can monitor metrics post-deployment, detect issues, and even trigger rollbacks.

The ultimate goal is to build trust in agents through comprehensive verification, mirroring human developer instincts (e.g., using Chrome DevTools). This allows humans to focus on higher-value tasks (why the work is done, overall strategy) while agents handle the minutiae of implementation and verification, leading to faster development cycles and more reliable software.

• Tags:
• Agentic AI
• Developer Tools
• Feature
• CI/CD
• Software Reliability
• Playwright
• LLM Judges
• Verification Loops
• Self-Healing Code
• Evaluation Baselines
• System One Models
• Outcome-Driven Development