Agent failures are unlike traditional software failures. This workshop provides a practical framework for monitoring production agents, moving beyond evals to real-world observability by using explicit signals (errors, latency) and implicit signals (user frustration, refusals, self-diagnostics) to catch regressions and understand agent behavior.
Patrick Debois argues that as AI coding agents become more capable, the context that drives them—prompts, rules, and memory—needs its own engineering discipline, akin to how we manage code. He introduces the Context Development Lifecycle (Generate, Evaluate, Distribute, and Observe) to make context a shared, repeatable, and improvable part of software delivery, creating a flywheel effect where better context leads to better agent output and continuous improvement.
Liam McGarrigle demonstrates how to build secure, observable, and controllable AI agents in n8n. The workshop covers creating a human-in-the-loop workflow for managing Gmail and Google Calendar, focusing on n8n's visual system for tool configuration, prompting strategies, and implementing essential approval steps to prevent unintended actions.
An evaluation platform is more than a simple test runner; it's a complex system for creating shared definitions of quality. This talk explores the evolution of eval platforms from basic spreadsheets to sophisticated, integrated systems, highlighting the hidden data and systems engineering challenges involved in making them credible, scalable, and usable for building trustworthy AI agents.
Yao Yue challenges the dominance of line charts in telemetry visualization, arguing that their time-centric nature and misleading interpolations often obscure insights. She proposes a 'form follows function' approach, advocating for visualizations tailored to the data's shape, the metric's type (e.g., step charts for counters, heatmaps for histograms), and the specific, often 'timeless,' operational questions being asked. The key is to transform time-series data into more flexible formats like panel data to derive clearer, more direct answers.
This talk by Juan Herreros Elorza explores how to design internal developer platforms for a future where AI coding agents are first-class users. It argues that the same best practices that make platforms accessible to humans—self-service interfaces, well-defined APIs, local-first workflows, and rich observability—are now critical prerequisites for agents to autonomously build, debug, and ship software. The session provides concrete principles for platform design, discusses how to manage AI-assisted contributions, and emphasizes the need to measure the impact of these changes on developer productivity and system reliability.