Multi Agent AI and Network Knowledge Graphs for Change — Ola Mabadeje, Cisco
A product manager from Cisco's incubation group, Outshift, details a solution that uses a multi-agent AI system combined with a dynamic network knowledge graph to solve critical issues in IT change management. The system integrates with ITSM tools like ServiceNow to automate impact assessment, test plan generation, and pre-production validation in a "digital twin" environment, significantly reducing production failures.
The project originated from a significant customer pain point: the high rate of failures in production environments following network change management processes. After analyzing the typical workflow, Cisco’s incubation group, Outshift, identified key stages where AI agents could provide substantial value, moving beyond simple rule-based automation to address the complexity and risks involved.
A Three-Pillar Architecture for Intelligent Change Management
The proposed solution is built on three core components designed to work in concert:
- Natural Language Interface: This serves as the primary interaction layer for both network engineers and external systems. The architecture is designed for system-to-system communication, enabling AI agents within an ITSM tool like ServiceNow to seamlessly communicate with agents on the solution side.
- Multi-Agent System: The core of the application features specialized agents, each tasked with a specific function. These agents handle impact assessments, test plan creation, and reasoning about potential failures. This division of labor allows for more focused and effective problem-solving.
- Network Knowledge Graph: This acts as a “digital twin” of the live production network. It combines a detailed knowledge graph with a suite of testing tools, providing a rich, dynamic context for the AI agents to operate within.
Deep Dive: The Network Knowledge Graph
Creating a useful digital twin of a complex network environment presents a significant challenge. Customer networks are heterogeneous, with a wide variety of vendors (firewalls, switches, routers) that produce data in disparate formats (YANG, JSON, streaming telemetry).
The primary goal was to create an ingestion pipeline that could unify this data into a consistent representation that AI agents could understand and act upon predictively. The key requirements for the knowledge graph technology were:
- Multimodal Flexibility: Ability to handle key-value pairs, JSON, and complex entity relationships.
- Performance: Near-instant access to node information for engineers querying the graph.
- Operational Flexibility: A schema that could be consolidated into a single, unified framework.
- RAG and Vector Support: Built-in vector indexing to enable future semantic search capabilities (Graph RAG).
- Ecosystem and Multi-Vendor Support: Easy integration into customer environments without heavy lifting.
After evaluating options like Neo4j and ArangoDB, the team chose ArangoDB, partly for historical reasons related to other internal projects. The resulting ingestion service performs an ETL process, transforming all incoming data into the OpenConfig schema. This networking-specific schema is well-documented online, making it well-understood by LLMs and ideal for agent interaction.
The graph itself is structured in layers (e.g., raw configuration, data plane, control plane). This allows agents to perform efficient queries by targeting only the specific layers relevant to their task, such as comparing configuration files or testing reachability.
Deep Dive: The Agentic Layer
The agents in this system are built using a framework that aligns with a broader, open-source vision called agency.org. This collective, supported by Cisco, LangChain, and others, aims to create open standards for agent interoperability, covering identity, skills definition, discovery, and observation.
Within this application, there are five primary agents built on ReAct reasoning loops:
- Assistant Agent: The central planner that orchestrates tasks among the other agents.
- Query Agent: The specialized agent responsible for all interactions with the network knowledge graph.
- Executor Agent: The agent that runs tests within the digital twin.
- (And two other specialized agents).
A key optimization was performed on the Query Agent. An initial attempt to use a standard RAG approach for querying the knowledge graph proved inefficient, consuming excessive tokens and time as it traversed all graph layers in a reasoning loop. To solve this, the team fine-tuned the agent using schema information and example queries. This led to a drastic reduction in both token consumption and query response time.
Workflow in Action: A Firewall Rule Change
A demo showcased the end-to-end workflow for a common task—implementing a firewall rule change to accommodate a new server.
- Ticket Creation: An engineer submits a natural language ticket in ServiceNow.
- Impact Assessment: The system ingests the ticket. The first agent summarizes the request and generates a detailed impact assessment, identifying potential downstream implications of the change. This report is then automatically attached back to the ServiceNow ticket.
- Test Plan Generation: An agent reasons about the intent described in the ticket and generates a comprehensive test plan. It outlines the specific test cases that must be run to validate the change and includes the expected results for each. This plan is also attached to the ticket for the approval board.
- Pre-Flight Testing: The proposed configuration change is pulled from a GitHub repository. The Executor Agent then:
- Takes a real-time snapshot of the network state from the knowledge graph.
- Runs all the test cases from the test plan within the digital twin, using tools like Batfish to simulate the change against the network snapshot.
- Reporting: After the tests are complete, the agent generates a final report detailing which tests passed or failed. For failed tests, it provides recommendations for remediation. This final report is attached to the ticket, giving the change approval board a complete, validated record before implementation.
Evaluation and Conclusion
The team emphasizes that for a system like this, extrinsic evaluation metrics are paramount. The focus is not on intrinsic graph properties but on quantifiable measures that map directly back to the customer’s use case and business value. The project is currently an MVP, but the key learning is clear: the combination of a dynamic network knowledge graph and an open framework for building AI agents is critical to creating a truly scalable and effective solution for modern network operations.