A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.
Cline's CEO explains why open-weight models are rapidly taking over coding-agent workloads, and why the agent harness, not raw model capability, determines reliability. The talk covers model-specific tuning and failure handling, context and token optimization, self-improving harness loops, an SDK-first and UI-agnostic architecture, interactive evaluation built from real-world usage data, and production inference challenges such as KV cache hit rates.
Rafal Wilinski of Runlayer argues that when one agent solves a hard problem, that knowledge should become a skill shared across the whole company. He covers why skills matter more as agents run longer, three problems with current skills, using MCP as a governed distribution layer, distilling skills from successful and failed agent traces inspired by the Voyager paper, and the organizational flywheel as a durable competitive advantage.
Siraj Gupta of Warp outlines three practical ways software factories can improve over time: self-updating skills driven by outer-loop agents with pull-request review, versioned and traceable persistent memory that works across harnesses, and cost-aware model routing with custom rules and evals.
A deep dive into the evolving world of LLM harnesses, demonstrating how they've transformed from simple prompt engineering into sophisticated self-improving systems. The discussion covers architectural advancements (e.g., Prime Agent's context as cache, Turing vs. Von Neumann), practical applications (auto-researchers, on-device personal AI like OpenJarvis, YC's QM for work), and the challenges and future directions of agentic AI. It emphasizes that harness innovation is key to unlocking significant performance gains and enabling LLMs to operate autonomously and adaptively.