Sandboxing, Agent Harnesses, and Agent Teamwork

Shahram Anver, Co-founder and CEO of Cleric, discusses the critical evolution of AI agents for Site Reliability Engineering (SRE), emphasizing that true value lies in operational memory and continuous learning, rather than merely fast alert triage. He outlines Cleric’s architectural journey, from an initially complex “ant colony” design in late 2023 with less capable models (GPT-3.5) to a simpler, more effective “query agent” model. The transition involved moving from rigid, custom tools to allowing agents to utilize bash and Python, a change facilitated by more advanced models like Claude Code. This shift, however, necessitated robust sandboxing to manage the increased non-determinism, reflecting a mental model of “as much non-determinism inside the box, but make a really strong box.”

Anver highlights that the core investigation agent itself has become the least important part of Cleric’s stack. The more durable and challenging problems now revolve around testing and verification across highly diverse customer environments, which often act like distinct, complex ecosystems where an agent, like a new engineer, needs to “ramp up” and learn local conventions. Unlike coding agents, which operate on a finite code base but can tackle infinite tasks, SRE agents face an infinite “search space” (a “needle in a haystack”) for a finite set of problem types (e.g., latency spikes, outages). This makes verification and learning paramount, especially for subtle, idiosyncratic issues, as opposed to binary, obvious outages, where user tolerance for error is extremely low.

The conversation delves into the evolving human-agent dynamic. Early agents suffered from low trust, but as capabilities have rapidly advanced, a new challenge emerges: humans become overly reliant, leading to a “spidey sense” problem where a “mostly right” agent can be more dangerous than one that is frequently wrong (as the latter prompts human verification). Anver posits that engineers will increasingly transition into roles as “managers” or “craftsmen” overseeing agents, focusing on “token maxing” – optimizing agent resource allocation – and setting clear “bounds” and “guidelines” for agent autonomy. This process externalizes implicit human operational knowledge, making it actionable for agents.

Cleric’s core value proposition lies in building this “operational memory” through continuous learning. Key data sources include Slack conversations (for historical context and common resolutions), code repositories, and Kubernetes infrastructure state (as the definitive “source of truth” against potentially outdated documentation). Agents learn from incidents, and their findings are presented to humans for verification and editing. Feedback mechanisms, such as “good boy/bad boy” ratings and explicit commands to “commit to memory” or “forget eternally,” help curate this knowledge. An intriguing concept is the “Investigation Auditor” sub-agent, which acts as a “Debbie Downer” to ensure thoroughness, sometimes prompting complaints from the main agent itself.

Looking to 2027, Anver envisions agents capable of self-healing infrastructure, detecting issues, reversing changes (e.g., from a coding agent), and reporting back to humans upon waking. This future necessitates addressing the “one-on-one problem” (humans setting policies for individual agents) and the “stand-up problem” (teams of agents collaborating effectively). Agent teams, where distinct agents communicate via protocols rather than shared context, are seen as crucial for specialized roles (e.g., coding agent interacting with an SRE agent to prevent future incidents). The ultimate goal of SRE is not merely to reduce MTTR but to prevent incidents by feeding production learnings back into the development cycle. Open standards for inter-agent communication (like Google’s A2A initiative) will be vital.

Finally, Anver discusses the future of software development itself, distinguishing between durable, complex systems (like BigQuery or payment processing) that demand high accuracy and rigorous maintenance, and lightweight “skills” or “vibe coded” solutions. The latter are bespoke, often local-first applications built quickly to address specific needs or high friction points with existing tools. While acknowledging the “dopamine hit” of productive side quests (vibe coding), he emphasizes the importance of founders staying focused on their “main quest,” as the definition of a “durable problem” in AI is constantly evolving.

• Tags:
• AI Agents
• Code Generation
• Large Language Models
• Devops
• Autonomous Systems
• MLOps
• Observability
• SRE
• Sandboxing
• Knowledge Graphs
• Agent Architectures
• Continuous Learning
• Production Operations
• Operational Memory
• Human-Agent Interaction
• Incident Management