Fighting AI with AI — Lawrence Jones, Incident

As AI systems become complex hierarchies of prompts, agents, and tools, debugging them manually becomes intractable. At Incident.io, the AI SRE product runs hundreds of prompts per investigation, analyzing logs, metrics, and traces. When it produces an incorrect root cause, a human engineer cannot feasibly trace the error through the thousands of steps. The core challenge is managing this complexity, which led to the realization that they needed to use AI to effectively build and debug their AI.

Managing Prompts with an Agent-Friendly eval Tool

For Incident.io, evals serve as unit tests for AI prompts. These are YAML files that live alongside the Go-based prompts and define a prompt, input data, and specific grading criteria to determine if the output passes or fails. This setup is crucial for verifying prompt changes before they are merged.

However, this approach faces challenges:

• Unmaintainable Test Data: Realistic evals often require entire incident reports as test data, making the YAML files massive (e.g., 2MB) and difficult for humans to work with.
• Agent Context Limits: Coding agents struggle to work with these large eval files, as they quickly exceed the context window limit, preventing effective modification.

To solve this, the team built eval tool, a small command-line interface (CLI) designed for agents. It allows an agent to interact with eval files programmatically—listing, editing, replacing, or adding test cases—without needing to load the entire file into its context.

This tool enables a “Red-Green Runbook” for coding agents to follow:

1. The agent first creates a new eval case that proves the existence of a bug (the “red” state).
2. It then modifies the prompt until the new eval passes (the “green” state).
3. Crucially, it runs the entire test suite to ensure the change hasn’t broken any existing evals (regression testing).
4. Finally, it attempts to consolidate and simplify the prompt to prevent it from becoming bloated and unmaintainable over time.

The Biggest Unlock: Debugging UIs as File Systems

While the eval tool is effective for changing a known prompt, the biggest challenge in a complex system is often identifying which prompt is the source of the problem. The AI SRE is not a single prompt but a deep hierarchy of over 10 agents and 50+ prompts and tools.

The team’s most significant breakthrough was serializing their debugging UIs into downloadable file systems. Instead of an agent trying to parse a UI, an engineer can now download the entire trace of a faulty interaction as a structured directory of text files.

The workflow is as follows:

1. When a bad user experience is reported, the engineer downloads the entire interaction as a file system.
2. This file system is dropped into a coding agent session (e.g., Claude Code).
3. The engineer describes the incorrect behavior in natural language (e.g., “The root cause analysis was wrong, it should have identified X”).
4. The agent can then use standard tools like grep to trace the logic through the self-documenting file structure, navigating the prompt hierarchy to pinpoint exactly where the reasoning went wrong.
5. Once the faulty prompt is identified, the agent can be instructed to use the red-green eval runbook to fix it, all within the same session.

Fleet-Scale Analysis with AI Pipelines

To understand performance at scale, the team runs daily “backtests” of thousands of investigations across hundreds of customer accounts. This produces a high-level metric, like “86% accurate RCA,” but fails to explain the “why” behind the number.

To get deeper insights, they developed an AI-driven analysis pipeline using a repository of structured playbooks called “scrapbook.”

1. Parallel Analysis: The pipeline begins by spinning up ~25 agents in parallel. Each agent is assigned a single investigation from the backtest and performs a detailed analysis based on a structured playbook, storing its findings in files.
2. Clustering and Synthesis: In the next stage, a meta-agent performs cohort clustering on the individual analysis files. It identifies common failure modes and systemic patterns, answering questions like, “Why is our system performing poorly for this specific customer?” or “What type of incident are we consistently misdiagnosing?”

This process produces a rich, actionable report that not only identifies failures but also suggests concrete steps for improvement. By combining this analysis with access to the codebase, the agent can even pinpoint the exact code that needs to be changed. This turns fleet-scale monitoring from a simple number into a powerful, automated system for continuous improvement.

• Tags:
• Prompt Engineering
• LLM
• Feature
• SRE
• AI Engineering
• Debugging
• AIOps
• Tooling