Always-on agents run production without the on-call tax — Justin Smith, Resolve AI

Introduction and The Shifting Bottleneck

Justin Smith, a founding product engineer at Resolve AI, introduces the concept of “always-on agents” designed to run production systems without incurring an “on-call tax.” With 15+ years in monitoring and observability, Smith highlights a significant industry shift: while the first wave of AI has drastically improved developer productivity, leading to faster code shipping and larger PRs, it has concurrently shifted the bottleneck from writing code to running code. Citing a survey, Smith notes that 70% of an engineer’s time is spent on operational tasks—maintaining platforms, scaling infrastructure, debugging incidents, on-call duties, hotfixes, and updating runbooks—rather than coding. This operational burden is exacerbated by AI, which introduces more changes and complexity into production, requiring a “full-stack AI” approach that focuses not just on models but on the context and domain-specific actions around them.

The Complexity of Modern Systems and Resolve AI’s Approach

Modern systems are inherently complex, involving multiple teams and systems with their own goals, all contributing to organizational objectives. Smith argues that managing this increased complexity necessitates the integration of AI within production environments. Resolve AI’s foundational hypothesis was that the influx of coding velocity driven by AI would lead to more issues, increasing the need for AI to operate and run these systems effectively. Resolve AI addresses this through various agent types.

Resolve AI’s Agent Capabilities

Resolve AI leverages a multi-agent architecture to tackle different operational challenges:

• On-call Agents: These agents perform triage and root cause investigation for alerts, significantly easing the burden on human on-call engineers who might be unfamiliar with recent changes or specific system intricacies. They can also support larger incident management activities, coordinating across teams and maintaining alignment during complex incidents.
• Background Agents: These agents handle the “long tail” of operational work that isn’t an immediate fire. This includes proactive monitoring, addressing one-off issues, and reducing cognitive load related to system health. Smith emphasizes that while models are capable, their success hinges on understanding the environment, how services interact, and where hotspots are. Resolve’s learning system continuously captures and evolves this knowledge as the system itself changes.

Core Principles of Background Agents

Resolve AI’s background agents are defined by three key principles:

1. When they run:
   • Schedule-based: For routine reports or ongoing summarizations (e.g., weekly on-call handover preparations).
   • Event stream-driven: Triggered by key events like deployments in CI/CD pipelines, or messages in platforms like Slack.
   • Message-based: Direct commands from users to perform specific tasks.
2. How they run:
   • Operate continuously in the cloud, independent of a user’s local machine.
   • Run within a sandboxed environment with its own file system, allowing self-organization.
   • Incorporate a learning loop and knowledge/memory system to understand the environment, reflect on tasks, and apply learned insights across different tasks, adapting as the system evolves.
3. How they know what to do:
   • Utilize a task system that integrates skills from other systems and various external integrations.

The Primacy of Production Context over Execution

A central tenet highlighted by Smith is the distinction between mere execution and understanding production context. While an execution engine can load a dashboard, it’s the production context that allows an agent to discern if a metric “smells wrong” even without a clear explanation. Building and maintaining this evolving knowledge system is where the true challenge and value lie, as it keeps pace with environments changing faster every month.

Practical Workloads for Background Agents

Smith elaborates on four key workloads where background agents excel:

1. Deployment Monitoring:
   • Agents monitor change events (e.g., GitHub release tags) to ensure system stability.
   • Unlike traditional CI/CD which provides baseline checks, Resolve’s agents analyze actual code changes to infer specific telemetry signals that would expose trouble for that particular change.
   • This allows for a customized monitoring plan; for instance, if a checkout service is replaced, the agent would watch checkout latency, error rates, and follow the causal chain into the Kafka pipeline.
   • Crucially, the timing for checks is not hardcoded; agents can decide to re-evaluate in an hour or even three days, acknowledging that certain issues surface intermittently.
   • This also covers changes not typically piped through CI/CD, such as feature flags or infrastructure modifications, which often lack monitoring until an alert fires.
2. Scheduled Health and Anomaly Checks:
   • Ongoing, periodic checks of system health, such as daily “morning reports” or monitoring specific third-party services after a related change.
   • Can be configured to watch a particular area for a defined period (e.g., a week) and then cease.
3. Operational Reports and Handoffs:
   • Summarize daily or weekly activities, investigations, and notable changes for on-call handovers or general team awareness.
   • These reports can be interactively refined; for example, a user can instruct the agent in Slack to “make it shorter” for future reports.
4. First Responder to Engineering Questions:
   • Agents passively watch critical Slack channels for engineering questions.
   • If the agent has sufficient confidence, it can directly answer questions.
   • If unsure, it can DM the human engineer for confirmation before responding publicly, showcasing emergent, intelligent behavior and flexible confidence management.

Integration and Future Vision

Resolve AI aims for a natural integration into existing workflows, primarily via platforms like Slack or MS Teams, making the agent’s interaction feel first-party. Users can configure new recurring tasks by simply conversing with the agent, which will explore the environment, ask clarifying questions, and set up the task. Resolve’s underlying agent harness, knowledge system, and ability to integrate external skills allow for deep customization and extensibility into any existing system.

Conclusion

Smith concludes by reiterating that the primary cost of operational work is not task execution itself, but navigating environmental complexity. Background agents offer a composable, flexible solution that runs on schedules or triggers, enabling exploration of diverse use cases. The interaction point with these agents is designed to be where engineers already work, minimizing disruption while maximizing the benefit of augmented intelligence in production operations.

• Tags:
• AI Agents
• Software Engineering
• Devops
• Autonomous Systems
• Contextual AI
• MLOps
• Observability
• SRE
• Operational Efficiency
• Incident Management
• Production Systems
• On-Call Management
• Slack Integration