How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Evan Kormos of Coinbase presents a practical framework for bringing AI agents into policy-driven, regulated workflows. He models processes as DAGs with loops inside each node, argues that a loop is only as trustworthy as its check, and shows how evals and traces create a new confidence mechanism. Agents then move through supervised, sampled, and exception-only stages.
Evan Kormos, a player-coach at Coinbase with 20 years of software experience (15 in enterprise financial services), starts from the idea that policy is the original system prompt. Policy is written with language like must, may, and should, and it sets the rules for how work gets done, especially for risk and compliance. He separates the build state, where policy is defined or changed, from the run state, where the system is constantly evolving through product-driven changes. Everything in the run state is still expected to stay roughly bound by policy.
Where agents fit
Using the familiar people, process, and technology model, Kormos treats agents as an operational model that cuts across all three. They work with people, execute parts of the process, and are built on technology. The talk focuses on where agents meet process: what work they are given, what boundaries they operate within, and how trust in their outputs is built.
Workflows as DAGs
He argues it is worth mapping workflows even when they seem basic, since approvals and verifications impose gated, ordered steps. Agents change what is worth parallelizing. Reviews that used to be too costly or complex to coordinate can now run independently and be merged, while the required checks stay in place. A directed acyclic graph (DAG) captures these steps and dependencies as a flow that moves forward without looping back. Apache Airflow is cited as open-source software that defines such DAGs in Python. Examples include commercial lending, listing a token on the exchange in a specific region, and regulated software delivery, which becomes the running example.
Inside a node: the loop
Inside a node, the work is a loop. A step is performed and checked, and if the result is not good enough, feedback is used to iterate until it passes. Kormos describes this as a directed cyclic graph (DCG) and points to LangChain’s LangGraph as a tool for managing state in such graphs. He is clear that this is an oversimplification, since real graphs can be very complex. The inner flow is where people and agents work together, while the outer flow is bounded by a policy boundary.
Exceptions, volume, and bottlenecks
Not everything goes to plan, so exception flows must be handled without letting work get stuck and without designing for every edge case. Examples include canceled code or code that fails security standards. Adding agents shifts volume across the flow. If code is reviewed faster, design and security scanning must keep pace. Kormos argues that operational visibility through metrics matters more than the authoring of the workflow tool, because parallelization and efficiency gains create new bottlenecks.
Quality inputs and the check
Variability in inputs and outputs is a core challenge in AI engineering. He believes higher-quality inputs produce better outputs at lower compute cost, and suggests injecting the DAG state into the agent so it understands where it fits in the bigger picture. This supports longer-horizon goals and makes it important to optimize the cost and efficiency of the loop. His central point is that the loop is only as trustworthy as the check. The hard part is defining what good looks like and having a reliable way to evaluate it, because more passes do not necessarily mean better results.
Evals and traces as a confidence mechanism
Kormos draws a parallel with traditional software, where unit tests and test cases build confidence that changes do not break other things. Agents are probabilistic, so the same request can yield different answers, and more than one answer may be acceptable. Exact output comparison is not enough. Teams need representative datasets, a definition of good, and repeated evaluation runs analyzed statistically across volume. Traces, which he describes as runtime logs of what happened, complement evals. Together they turn quality into evidence and statistics rather than a feeling about a model.
Which steps should an agent own?
Using lending as an example, with KYC, financials, credit memo, risk review, documentation, and decision all as nodes, he suggests asking whether each node can be checked against policy easily, can run in a bounded loop, can be traced, and can fall back to a human. If the answer is yes, the node may be well suited for automation. Still, AI assistance should be sought throughout the DAG, and the question is less whether AI is involved than whether it is expected to take over at some point. The decision node, which is essentially the policy check, should remain human-driven, and it is harder to hill climb.
Supervised, sampled, then exceptions only
Rollout happens in three stages. On day one, the process is supervised. Humans manually verify outputs, refine datasets, and work with domain subject matter experts to define good. When measured confidence reaches a policy threshold, the process moves to sampling, where humans review a slice of the output while systematic evaluation through traces ensures that unreviewed outputs still meet the quality bar. Once the confidence threshold is cleared, humans mostly handle exceptions while still acting as participants in refining datasets and ensuring quality.
Takeaways
The recipe is to keep the process deterministic on the outside and adaptive on the inside. Quality comes through evaluation, which builds a complementary confidence mechanism grounded in research and statistics, so trust in outputs is earned rather than assumed. Quality is a team sport: AI engineers own the pipeline that takes product signals from all directions and delivers well-built applications with MLOps behind the scenes. Finally, agents accelerate across the entire operating model. Kormos closes by noting that references are included on the final slide.