MCPs for Observability Stacks

Diana Todea, Head of Developer Relations at VictoriaMetrics, delves into the application of Model-Controller-Pair (MCP) servers within observability stacks to significantly enhance troubleshooting of production systems. She outlines what developers seek from observability: a unified view of metrics, logs, traces, and events for complete system context, proactive anomaly detection, automated operational tasks, reduced Mean Time To Resolution (MTTR), accelerated root cause analysis, faster troubleshooting with correlated telemetry, and ultimately, trustworthy and reliable data.

How MCPs Enhance Observability

MCPs leverage AI to transform traditional observability:

• Query and Correlation: They query and correlate telemetry across the entire observability stack.
• Automated Anomaly Detection: They automatically identify unusual behavior, moving beyond static thresholds. AI investigates these anomalies, retrieves relevant metrics, and guides root cause analysis.
• Natural Language Interaction: Complex query writing and manual dashboard navigation are replaced by natural language interactions.
• High-Quality Data: AI assists in validating anomaly scores, minimizing guesswork.

Understanding Anomalies and Anomaly Detection

An anomaly is defined as a data point or pattern that significantly deviates from expected behavior, manifesting as spikes, drops, or other unusual patterns inexplicable by normal fluctuations. These can stem from data collection errors, equipment malfunction, sudden behavioral shifts, or data drifts.

Anomaly detection is the process of identifying and highlighting these unusual data points or patterns within a given timeframe, crucial for engineers to prevent potential production issues.

Practical Demonstrations of MCPs

Diana presents several practical demonstrations:

1. Backend for Anomaly Exploration (VictoriaMetrics):

   • She shows a basic model detecting anomalies in error_spans from logs data.
   • The original data (Y) shows typical fluctuations, but the anomaly detection process introduces Yhat (model prediction) and Yhat_upper/Yhat_lower (business boundaries).
   • Anomalies are highlighted with red dots, emphasizing the importance of defining business boundaries for contextual analysis.
   • The system allows exporting model configurations and generating alert examples in YAML.

2. MCP Concepts and Skills:

   • MCPs connect an AI agent to live tools, APIs, documentation, and data, enabling the agent to perform actions rather than just explain.
   • Skills are reusable playbooks or recipes that guide the AI agent to achieve specific objectives. They are packed with recommended steps and guardrails.
   • MCPs are designed to help with troubleshooting, unified visibility of observability data, and querying documentation.
   • The proposed architecture includes a three-layer approach: traditional observability stacks, an anomaly detection layer, and the MCP server.

3. Two MCP Servers in Action with an AI Assistant:

   • Using a Cloud Desktop environment, two MCP servers are connected: one for the VictoriaMetrics Cloud observability stack and another for the VM Anomaly backend.
   • Skills like Query, Review, Configuration, and Alert Manager are defined to guide the AI assistant.
   • Metrics Overview: The AI assistant provides a summary of deployment metrics, including scrape details, ingestion rates, and resource usage.
   • Unused Metrics Identification: It identifies and helps clean up unused metrics (e.g., finding 98% of metrics were unused), promoting efficiency and proactive management.
   • Dashboard Creation: The system can generate dashboards on the fly from query results.
   • Anomaly Configuration & Alerting: The AI assistant, using the VM Anomaly configuration skill, suggests suitable models for anomaly detection and designs alerts in YAML format (as a draft, awaiting human verification).

4. AI Assistant Playground for Anomaly Detection:

   • A natural language interface allows users to ask for queries suitable for anomaly detection.
   • The assistant loads tools like get_server_queries and search_documentation to provide relevant suggestions.
   • Query Profiling: The AI assistant can profile a specific query’s time series characteristics (e.g., checkout_latency_percentile_95). It might identify that a current model (like profit) is a

• Tags:
• AI Agents
• AI
• Feature
• MCP
• Time Series
• Observability
• Machine Learning
• Metrics
• Anomaly Detection
• Natural Language Processing
• Skills
• Logs
• Troubleshooting
• VictoriaMetrics
• Alerting