From Spikes to Stories: AI-Augmented Troubleshooting in the Network Wild // Shraddha Yeole

In modern network operations, professionals are often overwhelmed by a flood of telemetry data from disparate dashboards, leading to cognitive overload and “contextual amnesia” during troubleshooting. This makes it difficult to quickly isolate the root cause of an issue, especially for those who are not deep network experts. The goal is to evolve from simple data presentation to intelligent data interpretation, enabling faster fault isolation and resolution.

From Dashboards to AI-Augmented Stories

The core solution is an AI agent designed to convert confusing spikes and complex visualizations into a coherent narrative. Instead of manually clicking through multiple layers of data—like HTTP, network, and BGP routing—a user can click a single “Explain” button.

This action triggers an AI assistant that analyzes and correlates metrics across all layers to deliver a concise fault domain assessment. For example, during a Google Cloud outage that affected Spotify, the agent could immediately identify that the application layer was impacted, showing reduced availability across multiple locations and pinpointing specific HTTP error codes, thus saving valuable time in the diagnostic process. This approach significantly improves the mean time to identify (MTTI) and resolve (MTTR).

Technical Architecture and Implementation

The AI agent’s workflow is orchestrated through a sequence of modern tools and frameworks:

1. The user’s request is routed through Microsoft’s Semantic Kernel.
2. The kernel invokes a proprietary Views Agent, which calls internal API endpoints to gather relevant telemetry data.
3. Crucially, a pre-processing and enrichment step occurs before the data is sent to the Large Language Model (LLM). This step is vital for managing data complexity and volume.
4. The processed data is then passed to the LLM, which is currently Anthropic’s Claude 3.7 Sonnet model, chosen for its strong reasoning capabilities.
5. The LLM’s response is formatted and displayed to the user, with chat history cached for faster subsequent explanations.

Prompt Engineering and Evaluation Strategies

Crafting effective prompts was a key focus. The team employed several advanced techniques:

• Role Prompting: The prompt assigns the LLM the persona of an expert network engineer to guide its analysis.
• Chain-of-Thought (CoT) Reasoning: Developed in close collaboration with subject matter experts (SMEs), this technique instructs the model to perform a step-by-step analysis, mimicking an expert’s diagnostic process.
• Few-Shot Prompting: Examples of desired input-output formats were included to ensure the model’s responses are structured and consistent.

To ensure the reliability of the LLM’s output, specific structural guidelines were implemented:

• XML Tags: Using tags like </instructions> and </examples> helped the model better understand the prompt structure.
• JSON Schema: A JSON schema was provided in the system prompt to enforce a predictable output format.
• Data Model Annotations: Similar to Pydantic models, these annotations helped the LLM understand the semantics of the input data fields.

Evaluation is a multi-faceted process, combining human expertise with automated checks. It includes manual evaluations by SMEs using ground-truth data from well-known outages, alongside using an “LLM-as-a-judge” to programmatically track metrics like hallucination rates, answer relevancy, and prompt alignment.

Challenges and Future Directions

The primary challenge lies in the sheer volume and complexity of network and application telemetry. Sending raw API data directly to the LLM resulted in high token counts and weak correlations. This necessitated the development of robust pre-processing and data enrichment pipelines, including baselining metrics to provide the LLM with meaningful context.

Future work is focused on two main areas:

1. Continuous Learning: Implementing a human-in-the-loop feedback system to continuously refine the agent’s accuracy using ground-truth outage data and SME reviews.
2. Model Fine-Tuning: Moving beyond general-purpose models to fine-tune a model specifically on proprietary network data, which would then be hosted within their own environment for improved performance and specialization.

• Tags:
• Prompt Engineering
• LLM
• Anthropic
• Observability
• Agentic Systems
• AIOps
• Network Monitoring