We Trust AI Agents That Break Every Day - Shy Ruparel, Temporal
Shy Ruparel of Temporal breaks down the four gaps keeping AI agents from production, drawing on survey data from 554 engineers, and demos a crash-resilient $2,500 payout workflow built on durable execution with a local model.
Agents are now everywhere, but the distance between using them and running them reliably in production is where most teams stall. Temporal’s State of Development survey of 554 engineers and engineering leaders, now in its second year, puts numbers on four gaps between experimentation and production.
Usage versus production
Daily agent use nearly doubled, from 47.3% in the 2025 edition of the survey to 80.8% today. Yet only 49.1% of respondents say they run agents in production or that agents are core to how they ship. That leaves roughly a thirty-point gap between people who use agents every day and people who actually ship them. Something keeps breaking on the path to production.
Trust versus reality
Trust in agent output is high: 85.5% trust it at least somewhat, and about a quarter trust it almost completely. Failures are just as common. 41.1% of respondents hit agent issues daily or more, and 16.4% hit them every hour. Both facts hold at once. Teams that call themselves successful are 1.7 times more likely to hit issues continuously, and agent count shows no correlation with issue frequency, which suggests more mature teams may simply be looking harder for problems. The implication is that failure cannot be skilled away. The goal is not zero failures but making failures cost nothing. Security was cited by 40% of teams as the main blocker to autonomous agents, a topic that deserves its own treatment.
Prototype versus enterprise
51.3% of teams can take an AI prototype to production-ready code in hours, which is startup speed. That speed collapses in large organizations. At companies with more than 5,000 employees, about one in five report it takes months, compared with roughly one in twenty elsewhere. In regulated sectors such as banking, insurance, and payments, that time likely goes into approvals, audit trails, and guarantees that no payment gets lost. The survey does not break this down, but the process overhead is aimed at reliability.
Seeing versus recovering
When asked what limits productivity, respondents ranked tracking state (35.7%) and debugging (30.9%) far above model quality. 22.2% lose progress and redo work after crashes and restarts, and state management, which was rated fragile a year earlier, is now the top blocker. When an agent breaks, the most common place engineers turn for help is YouTube, a sign of systems that cannot explain what happened. Observability can tell you an agent failed, but reliability means the failure does not cost you anything. Only 13% are very confident they can observe and debug AI workflows at scale. Today, recovery often means someone reading a trace at 2 a.m. and repairing state by hand, which is not a realistic operating model.
A payout that survives a crash
The demo runs a nine-stage payout workflow for a synthetic $2,500 transfer on a local Qwen model. The steps include validating the request, reserving funds in an external bank system, having an agent investigate the transaction and draft a brief for review, and waiting for a human approval. The worker is then killed on purpose, and the approval is sent while no worker is running.
The approach behind this is durable execution: code runs to completion even when the process running it does not. Every step is recorded in an event history. Workflow code is deterministic and decides what happens next. Anything non-deterministic or side-effecting, such as LLM calls, tool calls, and bank API calls, runs as an activity whose result is recorded. The human approval arrives as a signal, which is captured in history even with no worker running. When a new worker starts, it resumes from the last successful event. Completed LLM calls are not repeated, so tokens are not spent twice, and only an activity that was mid-flight gets retried.
The resulting history shows each AI activity ran exactly once, and the bank submission completed on a replacement worker, with nothing redone from the beginning. The demo addresses seven of the blockers respondents named: state, debugging, cost, retries, approval, long-running work, and crash recovery. State is simply ordinary variables, with no custom database or cache. Retries are declared once as a policy rather than written as loops. If the model had failed, the payout would have proceeded without the brief, so a model outage degrades only the draft. The approval timer could wait a year, and the code would be the same. No crash-recovery code was written, because it comes from the platform. Debugging uses the same history the worker replays, which collapses observability and recovery into one record and closes the fourth gap.
This is already in production. OpenAI’s Codex runs on Temporal, and Replit runs its agents on it, with its engineering team noting that failing catastrophically deep into a task is a poor user experience. Bad model output is treated as an activity failure and retried under the same policy.