A Q&A from the final part of Anthropic's Claude Managed Agents workshop, covering CI/CD for agent configs in YAML with the ant CLI, eval practices, workspace-based resource sharing and tenant isolation, agent-driven code review, GPU-enabled self-hosted sandboxes, multiplayer Slack agents, long-running sessions, and the coordinator pattern for custom tools.
Kamalakannan Nandagopal from Postman explains why coding agents struggle beyond a single repo and how an API context graph, grounded in code and production telemetry, gives agents system-wide awareness. The talk covers evals built from real PRs, wins on API discovery, redesign and impact assessment, a stale-graph failure, and a 21-page architecture review generated by the agent.
Harrison Chase, co-founder of LangChain, delves into the critical role of the 'harness' in LLM agents, explaining how it orchestrates models and context. He covers customizing harnesses with middleware and sub-agents, the trade-offs between off-the-shelf and custom solutions for in- and out-of-distribution tasks, and the importance of evaluations and observability for continuous agent improvement. The discussion culminates in the 'data flywheel' concept and the `LangSmith Engine` for automating agent intelligence through iterative refinement.
Robert from OpenAI discusses the critical role of structured evaluations (evals) and graders for developing advanced collaborative agents. He explores the limitations of 'vibe-based' assessments, introduces a maturity model for evals, and presents a comprehensive rubric for measuring agent performance beyond simple accuracy, connecting these concepts to the power of Reinforcement Fine-Tuning (RFT).