Production monitoring for AI applications using W&B Weave

The Challenge of Production AI

Developing a high-performing AI application requires rigorous end-to-end evaluation across multiple dimensions like accuracy, cost, latency, and safety. However, even with comprehensive test datasets, it’s impossible to anticipate every real-world scenario an application will encounter in production. The non-deterministic nature of LLMs, corner cases missing from evaluation data, and unexpected changes in the production environment make continuous monitoring after deployment a critical necessity. The process of iteration and optimization does not end once an application goes live; it must be a continuous cycle.

Weave Monitors vs. Guardrails

W&B Weave provides two distinct mechanisms for managing application behavior in production: Guardrails and Monitors. Understanding their differences is key to implementing an effective monitoring strategy.

• Guardrails act as a synchronous, preventative barrier. They are an additional layer of protection designed to safeguard an application by blocking malicious or dangerous requests and responses before they reach their target. They are used for rigid enforcement of rules where certain behaviors are unacceptable.

• Monitors operate asynchronously, recording performance metrics in real-time without interfering with the application’s operation or blocking responses. This makes them ideal for tracking quality metrics over time using “LLM-as-a-judge” scores. Data collected by monitors is invaluable for detecting data or model drift, triggering alerts when metrics cross a threshold, and building new, more robust evaluation datasets.

Building a Monitor for Online Evaluations

The video demonstrates how to set up a “friendliness” monitor for a retail customer support chatbot to ensure customers receive respectful and helpful interactions.

The configuration process within the Weave UI involves several steps:

1. Assignment: The monitor is assigned to a specific operation within the application, such as the retail_support_agent.predict function that generates the customer response. Filters can be added to trigger the monitor only under specific circumstances, and a sampling rate can be set to control how frequently it fires.
2. LLM-as-a-Judge Score Definition: A score is defined to evaluate the output. This involves:
   • Naming the Score: e.g., “friendliness_score”.
   • Selecting a Model: Choose from models configured in the Weave account or open-source models hosted by CoreWeave with W&B Inference (e.g., gpt-4o-mini).
   • System Prompt: This provides instructions to the judge model. For the friendliness monitor, the prompt asks the model to “rate the chatbot’s friendliness on a scale from one (very unfriendly or cold) to five (exceptionally friendly and warm)” and to justify the score.
   • Response Format: Specify the desired output structure, such as JSON.
   • Scoring Prompt: This prompt reiterates instructions and includes a variable that will be replaced by the actual output from the support agent, feeding it to the judge model for scoring.

The Iteration Loop in Production

The power of online evaluations is in enabling a continuous feedback and improvement loop, even after deployment.

1. Initial Interaction: A user asks the chatbot, “What’s your return policy for laptops?” The chatbot provides a factually correct but terse response.
2. Evaluation in Weave: The interaction is captured as a trace, which triggers the friendliness monitor. The monitor’s LLM-as-a-judge scores the response a “3”, noting it was neutral but could be warmer. While not offensive, this falls short of the goal of a “4” or above.
3. Iteration and Improvement: This real-time feedback indicates a need for optimization. A good starting point is to revise the support agent’s system prompt to explicitly instruct it to be more friendly and welcoming.
4. Validation: After updating the prompt, the same user query is tested again. The new response is noticeably warmer. The Weave monitor confirms this improvement, assigning the new response a score of “4” and providing a reason that indicates the new prompt is effective.

Tracking Performance Trends

To maintain quality over the long term, it’s essential to track metrics beyond individual interactions. Weave Trace Plots allow for the visualization of monitor scores over time. By creating a line chart of the average friendliness score, teams can easily track performance, spot trends, and detect any degradation or drift in the application’s behavior, ensuring a consistently high-quality user experience.

• Tags:
• Evaluation
• Feature
• LLMOps
• Production AI
• LLM as a Judge
• Monitoring
• W&B Weave