Nicholas Arcolano of Jellyfish uses data from about 50,000 engineers across 447 companies to test what really drives AI coding output. A natural experiment forcing users onto Grok barely moved merged PRs. The analysis breaks throughput into tool, skill use, token consumption, company and team, and individual engineer, finding the individual and organization matter far more than the model.
Pepijn Senders, a principal engineer and former AI enablement lead at HelloFresh, explains how his team rebuilt a banking platform in six months with about 800,000 lines of code, mostly written by Sonnet, and why choosing a model that keeps developers involved matters more than raw autonomy. He covers measuring AI ROI through token spend versus output, keeping humans in control of code, the risks of agent recommendations, team refinement sessions, prompt word effects, and a meta harness built on Claude Code.
Ford Prior, a senior staff engineer and tech evangelist in GEICO's claims technology group, describes an AI-native multiplayer war room built to unblock major, outer-loop software changes. The talk covers a one-day, $1,000 token experiment across rentals, damages, and salvage legacy domains, the FleetKit markdown rulebook that steers the agent fleet, the worksheets and design artifacts it produced, and the cost, model-selection, and tooling lessons that shape what comes next.
A financial agent produces confident investment advice with fabricated details, redundant tool calls and bad summaries. This talk shows how traces, evaluations, and a coding agent connected through the Arize CLI and skills can investigate those failures, propose fixes, and test them with experiments. It covers why evaluations matter over vibes-based shipping, how judges and costs fit in, and why human review and application-specific evals are still essential.
Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.
Michael Patterson of Coder explains Simon Willison's lethal trifecta: private data access, exposure to untrusted content, and external communication, and why coding agents running on developer laptops hit all three. He covers prompt injection, the organizational blockers to enterprise agent rollout such as shadow AI and alert fatigue, and three architectural guardrails: cloud development environments, a model proxy, and an agent firewall.