Power agents with full context of your experiments and traces with W&B MCP server

The Weights & Biases (W&B) Model Context Protocol (MCP) is presented as a novel, fully hosted server designed to standardize and streamline how AI agents interact with W&B data and experiments. Initially an open-source repository, the MCP has evolved into a production-ready solution, now automatically provided with all W&B deployment types, including SaaS, dedicated, and on-prem instances. This protocol enables agents to directly access and act upon diverse W&B data assets, such as runs, traces, evaluations, artifacts, and reports.

Understanding the Model Context Protocol (MCP)

At its core, MCP provides a standardized format that allows agents to interact with a specific set of tools for communicating with W&B experiments and traces. This facilitates deep integration with various client applications, including:

• IDEs: Cursor, VS Code
• Coding Agents: CodeX, Gemini CLI
• Chat Agents: Mistral Chat, Cloud Desktop

Users have two main deployment options:

• Hosted Version: Automatically included with any W&B deployment, requiring no additional setup.
• Local Installation: For those who prefer local control, the MCP can still be installed and run locally from its public repository.

Installation and Configuration

Installation is designed to be straightforward:

1. Specify the W&B host URL.
2. Execute a single-line command in the terminal, incorporating your W&B API key.
3. For IDEs like Cursor, a dedicated button simplifies the process, pre-installing the necessary configuration and prompting only for the API key.

Overview of MCP Tools and Features

The MCP server offers a comprehensive array of tools, categorized to support various aspects of ML workflow management:

• Discovery Tools: These are crucial for intelligent context management. Agents can independently discover teams, projects, data within projects, and the schema of experiments. This capability enables smarter queries and allows agents to self-correct or refine their understanding of the context.
• W&B Model Interaction: Tools to query run-specific information, particularly valuable for managing and versioning training data within W&B Models.
• Experiment Management: Features designed to compare and diagnose training runs, often utilized by specialized training agents.
• Trace Analysis: Capabilities to query, count, and summarize traces and evaluations.
• Reporting: Tools for generating and analyzing reports, including a smart feature that allows for report analysis even before the final report is created.
• Object Versioning: Agents can list and manage versioned objects within W&B.
• Documentation Access: A built-in tool allows agents to retrieve information directly from W&B documentation for clarification or extended knowledge.

Demo: Hiring Agent Correlation Analysis

This demonstration illustrates MCP’s advanced data analysis and self-discovery capabilities within a hypothetical “hiring agent” project. The agent’s function is to process job applications and positions to make interview decisions, while also detecting potential hallucinations, leveraging a fine-tuned model for decision-making.

Problem Statement: Determine the correlation between a “quality score” and “age versus experience” metrics within the fine-tuning dataset.

MCP’s Analytical Workflow:

1. Initial Self-Discovery: Upon receiving a broad query, the MCP agent first employed its discovery tools to comprehend the project’s structure.
2. Schema Inspection: It intelligently examined various data types (traces, runs, artifacts) and their respective schemas. It correctly identified that the relevant dataset information was stored within “run summary tables,” after initially exploring traces.
3. Targeted Run Identification: The agent then pinpointed specific fine-tuning runs that had logged the necessary dataset, prioritizing visionary-win-10 due to its larger population size.
4. Automated Script Generation and Analysis: The MCP autonomously generated and executed a script to perform the correlation analysis, providing specific metrics and confirming the statistical significance of the chosen run. This showcased MCP’s ability to handle underspecified queries and perform deep, context-aware analysis.

Demo: RL Email Agent Training and Reward Regression

This demo focuses on optimizing an email agent designed to search inboxes and provide answers, utilizing W&B training with a serverless GRPO endpoint. The optimization process involves various ablation studies, such as different fine-tuning warm-up strategies, GRPO configurations, and reward settings.

Problem Statement: Compare the two most recent training runs to identify any regressions in reward metrics.

MCP’s Comparative Analysis Workflow:

1. Self-Discovery of Relevant Runs: The agent intelligently probed the project to identify the specific training runs and their associated rewards. It accessed historical data and, if necessary, prompted the user for clarification regarding ambiguous runs (e.g., requesting specific run IDs).
2. Utilizing the Comparison Tool: Employing a built-in comparison tool, the MCP analyzed the two specified Reinforcement Learning (RL) runs.
3. Regression Identification: It provided a clear overview of the reward changes, explicitly highlighting any regressions. For example, it accurately identified a regression in the “ruler score” from AAA (0.32) to SC5 (0.31), which was verifiable from the W&B charts. This demonstrates MCP’s efficiency in comparing experimental outcomes and pinpointing performance shifts.

Demo: Mistral Chat Integration and Mobile Workflows

This segment highlights MCP’s integration with conversational AI agents, exemplified by Mistral Chat, extending W&B’s capabilities to mobile and browser-based interactions.

Integration Process: Users can easily add W&B as a connector within Mistral Chat, providing their API key to gain access to all MCP tools.

Key Use Cases:

1. On-the-Go Status Updates: From a mobile device, users can query the status of ongoing runs within a project (e.g., the hiring agent project). The agent successfully identified the latest run (skilled-silence), noting its 0% evaluation, which indicated a probable crash or unsuccessful completion. This efficiently surfaced critical information remotely.
2. Automated Report Generation: Users can instruct the agent with commands such as, “Summarize the key performance metrics for my manager in a Weights & Biases report.” The MCP automatically generates a comprehensive W&B report, including a title, overview, index, summary of relevant runs, and analyses tailored for managerial review. This functionality is highly adaptable, supporting various workflow requirements and capable of leveraging additional “skills” alongside MCP.

Conclusion

The Weights & Biases MCP emerges as a powerful, hosted solution offering 20 integrated tools, accessible across a diverse range of client applications. It significantly simplifies AI agent development and model training by providing a standardized, intelligent interface for interacting with W&B data, supporting complex queries, self-discovery mechanisms, and automated reporting.

• Tags:
• AI Agents
• Generative AI
• Model Context Protocol
• MCP
• MLOps
• Model Evaluation
• Data Analysis
• Weights & Biases
• Experiment Tracking
• Tooling
• LLM Tools
• Machine Learning Operations
• Chatbots
• Mistral AI
• Run Comparison
• Automated Reporting
• Self-Discovery