Big updates to mlflow 3.0

Eric Peter and Corey Zumar from Databricks explain how MLflow is evolving to address the primary challenge in Generative AI development: ensuring and improving application quality. They draw an analogy to the traditional software development life cycle (SDLC), which relies on testing, QA, and observability, but note that GenAI’s non-deterministic nature and free-form text I/O make direct application of these principles difficult.

A Structured Lifecycle for GenAI with MLflow

MLflow provides a suite of features to bring a structured, iterative process to GenAI development, helping teams manage the complexities of quality assurance.

Tracing: Understanding What’s Happening

The first step to improving quality is visibility. MLflow Tracing allows developers to instrument their applications with a single line of code to capture the entire execution flow. This includes:

• Retrieving data (e.g., from a CRM).
• Intermediate data processing steps.
• The final calls to the LLM.

Crucially, traces can also capture and display end-user feedback directly in the UI, linking application performance to user satisfaction.

Evaluation: Defining and Measuring Quality with LLM Judges

Once an application is instrumented, the next step is to systematically evaluate its output. MLflow allows developers to create custom evaluators, often in the form of “LLM Judges.”

• Business-Defined Criteria: Developers can work with business stakeholders to define quality criteria in plain language (e.g., “meeting discussions must be summarized with the same sentiment”).
• SDK Implementation: These criteria are passed into MLflow’s SDK, which then runs the LLM judge across traces to score the application’s output on metrics like accuracy, relevance, or groundedness.
• Root Cause Analysis: The UI provides detailed justifications from the judge, explaining why a particular output failed a quality check, which helps developers quickly diagnose issues like model hallucinations.

Iteration and Monitoring

With a systematic evaluation process, developers can iterate rapidly.

• Offline Comparison: MLflow allows for the creation of evaluation datasets from problematic traces. Developers can run experiments with different prompts or models against this dataset and use the UI to compare results side-by-side, visualizing the impact of their changes on quality metrics.
• Online Monitoring: The same LLM judges developed for offline evaluation can be registered and deployed for continuous monitoring of live production traffic, ensuring that quality does not degrade over time.

The Next Step: Automating Quality Improvement with Agentic Insights

While the above tools provide a robust framework for manual quality iteration, the process of finding, diagnosing, and prioritizing issues is still time-consuming. The latest advancements in MLflow, called Agentic Insights, aim to solve this by using agents to analyze and fix other agents.

From LLM Judges to “Agent as a Judge”

Traditional LLM judges are limited because they typically only see a small part of the trace (e.g., the final input and output). The new approach uses an “Agent as a Judge”:

• Holistic Analysis: This agentic judge can read and navigate the entire trace, just as a human developer would. It can dynamically inspect tool calls, intermediate steps, and error messages to get a complete picture of what went wrong.
• Higher-Level Instructions: This allows for more open-ended and powerful evaluation prompts, such as, “Analyze this trace and determine whether tools were called properly.”

Automated Root Cause Analysis and Prioritization

The most powerful feature of Agentic Insights is its ability to automate the analysis of entire experiments. By giving a coding agent access to all MLflow traces, it can perform a comprehensive quality audit.

1. Understand the Goal: The agent first analyzes a batch of traces to understand the application’s purpose without needing to read the source code.
2. Generate and Test Hypotheses: It identifies a failure in one trace (e.g., a malformed SQL query) and forms a hypothesis that this might be a widespread issue. It then tests this hypothesis by searching all other traces for similar failures.
3. Stack-Rank Issues: The agent iteratively builds and tests hypotheses for various problems, ultimately creating a stack-ranked list of the most critical issues, quantified by their impact (e.g., “Tool call failures account for 40% of all errors”).
4. Generate an Actionable Report: The final output is a detailed report that summarizes the key issues, provides links to specific traces as evidence, and gives a prioritized list of recommendations. This report can be shared with a development team or even fed to another coding agent to automatically generate pull requests for the fixes.

These new capabilities, set to be released in MLflow 3.4, represent a significant step forward in automating the GenAI development lifecycle, allowing teams to deliver higher-quality applications much faster.

• Tags:
• Generative AI
• Feature
• LLM Evaluation
• AI Quality
• MLOps
• MLflow
• Agentic Insights