Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic
A Q&A from the final part of Anthropic's Claude Managed Agents workshop, covering CI/CD for agent configs in YAML with the ant CLI, eval practices, workspace-based resource sharing and tenant isolation, agent-driven code review, GPU-enabled self-hosted sandboxes, multiplayer Slack agents, long-running sessions, and the coordinator pattern for custom tools.
Agent definitions are treated like infrastructure. The ant CLI wraps the Claude Managed Agents API and works in a Terraform-like way: long-lived resources are declared in YAML, checked into the codebase, and pushed to the platform by CI/CD on every change. Anthropic’s own teams build durable resources the same way.
Messaging during custom tool waits
Execution pauses while the model waits for a custom tool result, so the platform blocks incoming messages at that point to preserve ordering. Messages sent while the model is doing other work are queued. Sending a message during a pending custom tool result returns a 400.
Evals and quality
Anthropic runs extensive evals whenever new models and capabilities arrive, since it has access to models before customers do. As a managed offering, it aims to revisit its harness assumptions as early as possible so that scaffolding no longer needed can be removed. Internal tooling runs different datasets across sessions, and because everything runs in the cloud, it can launch something like 10,000 sessions in parallel, then score results and tune prompts and assumptions. The work covers general purposes, not only coding.
For customers building their own agents, the main advice is to invest in high-quality evaluation data, since garbage in means garbage out. Early eval ideas were sometimes ill-defined and produced scores of 100% without clearly meaning anything about performance. Historical sessions are a practical source for building a dataset: review past results, score them, and see how the model is doing.
Sharing resources within an organization
Resources such as agents, memory stores and MCP channels are bound to a workspace, and any API key in that workspace can access them. The org admin creates workspaces and decides who gets access to which, so teams should design workspaces around their access-control boundaries. The platform has no fine-grained sharing controls on individual resources, but builders can expose partial access granularly through their own integrations so end users see only part of an agent.
Reviewing agent-written code
Generating code with many agents has shifted the bottleneck to review and merging. Anthropic’s answer is to integrate Claude Managed Agents with pull requests: each PR kicks off one or many sessions with subagents that evaluate changes and look for bugs. Internally this reduced per-PR review time and bug counts while keeping quality high. The harder open question is how to keep up with the much larger number of PRs that agents can now produce.
Review can iterate in a loop until comments are resolved. Beyond static review, an application can be deployed to a per-branch environment or run inside the sandbox so Claude can validate its own changes and act as a QA agent. Today there is no tunneling into those sandboxes, so Claude would drive a browser itself, validate the behavior, and possibly return a video.
Tenant isolation
Isolation is configured through the platform’s hierarchy. Teams can define an agent per customer, a workspace per customer for true isolation, or an org per customer, and they should maintain the mapping from customer to agent. On-prem deployment is not supported yet.
When one agent serves several customers, the definition can live in YAML in code, and CI tooling can synchronize it to multiple workspaces so one update propagates everywhere. Alternatively, all customers can share one organization or workspace, with each session tagged through a metadata field such as a customer ID. Where data segregation is a legal requirement, the answer is one workspace per customer, since resources are allocated per workspace, with synchronization tooling to keep their definitions aligned.
Custom compute, GPUs and memory
The default sandboxes are all identical, so workloads needing GPUs or more memory should use self-hosted sandboxes. Providers such as Modal offer GPU capacity, and first-party cookbooks show how to run the agent loop inside Modal, EC2, Daytona or other providers instead of the provided sandboxes. The environment specifications are documented, along with these integration guides.
Multiplayer agents in chat
A Slack-style multiplayer agent can map each thread to a session. Each incoming user message is injected with the sender’s name so the agent knows who is speaking, and anyone can message the agent independently. Memory can be shared across sessions so the agent learns from everyone. The agent should post through a Slack tool it calls explicitly, rather than mirroring every response. A Slack-channel-style design would use one agent per channel, with separate memory and access limited to that channel’s messages.
Very long-running sessions
No special configuration is needed for sessions running beyond six hours. The outcomes feature is the recommended approach for long tasks: define exit criteria, and the agent keeps iterating until it meets the rubric. Progress can be followed through the session event stream and the events API under the session. Alternatively, a custom tool such as an update-status tool lets Claude report progress at intervals it chooses.
The coordinator pattern for custom tools
To keep the main agent responsive while custom tools are pending, a coordinator agent can delegate that work to a subagent that defines and handles the custom tool. Agent A can talk to agents B and C, and the subagent emits the custom tool use. The subagent runs on a separate thread within the same core session, but the caller interacts only with the main thread, and the platform routes results by ID. The main thread can therefore sit idle while subagents are still working or waiting on tool results. This makes the pattern well suited to an agent that must always be available for conversation.