Aparna Dhinakaran of Arize AI argues that the AI engineer's job is shifting from writing features to building the loop that builds them: evals that review non-deterministic work, telemetry that agents consume, and always-on agents that act on it. She uses Arize's Signal agent as a case study, covering triggers, sandboxes, durable state, model and harness independence, and evaluating the evaluators.
Stephanie Jarmak, who moved from astronomy into building coding benchmarks and agent evals, argues that most agent evaluations are easy to misread. She covers how harness choice can swamp model differences, why public benchmarks serve shortlisting rather than decisions, how to design experiment-style evals with realistic tasks, repeated runs, and full traces, and how to set quality gates, manage human review load, verify agent records, and treat production as the final test.
Gareth Yoder of Vanguard describes modernizing a cash ledger handling up to 30 million calls per day by running data, producer, and consumer migrations in parallel. Key themes include exposing legacy code to AI through an MCP server, reducing a seven-day research cycle to fifteen minutes, reimagining rather than lifting and shifting, and maintaining observability throughout the migration.
Grepr CEO Jad Naous talks with a16z's Martin Casado about how coding agents have changed a 10-person engineering team, with projects that once took six months now finishing in about ten days. They examine whether AI can reason about and troubleshoot production distributed systems, why operations is a harder problem than code generation, and where AI adoption is likely to go next.
Sam, CTO at Gravity, argues that AI agent governance must be enforced in infrastructure rather than in prompts or policy documents. Through five live demos against a hotel-booking agent, he shows a gateway blocking destructive tool calls, preventing data exfiltration, rate-limiting runaway agents, serving cached semantic responses, and rejecting oversized prompts before any LLM call. He also covers observability, vendor decoupling via LLM routing, trimming bloated MCP tool lists, and managing shadow AI.
Matt Gibiec of Dynatrace argues that coding agents are fast but blind without real context about how services run in production, and introduces Bluebox, an observability agent that brings dependency maps, production baselines, and change history into local testing, issue detection, fix suggestions, and pre-build planning, with a human approving the final result.