MLflow 3.0: The Future of AI Agents

Eric Peter from Databricks presents a clear distinction between the traditional Machine Learning Operations (MLOps) lifecycle and the emerging lifecycle for AI agents, which he terms “Agent Ops.” While the core workflow of development, evaluation, and deployment remains, the nature of Generative AI introduces significant changes and challenges.

The Evolution from MLOps to Agent Ops

The shift to building AI agents has fundamentally altered the development process in several key ways:

• New Participants: The field is no longer exclusive to data scientists. Software engineers without deep ML experience are now key builders of these applications.
• From Training to Tuning: The classic model training step has been largely replaced by a more nuanced process of “tuning” or “prompting.” This involves integrating tools, managing function calls, and deeply understanding the application’s internal workings through tracing.
• Human-Centric Evaluation: Evaluation is no longer about checking a numerical prediction. Outputs are free-form language that requires human domain experts to assess quality, relevance, and safety. Traditional software testing frameworks built for deterministic outputs are insufficient.
• Deployment Complexities: Deploying agents requires new considerations, such as implementing robust guardrails to manage model behavior and ensuring comprehensive tracing and observability to understand what’s happening in production.

Core Pillars for Effective Agent Development

To successfully navigate this new landscape, two main categories of capabilities are required: GenAI-native evaluations and a robust Agent Ops platform.

1. GenAI-Native Evaluation

Evaluating generative models is uniquely challenging because both the inputs and outputs are free-form and non-deterministic. A user can ask the same question in multiple ways, and a model can provide semantically identical but textually different answers. An effective evaluation system must therefore include:

• Rapid Debugging: Tools to trace and visualize the complex flow of an agent’s execution.
• Human Feedback Collection: A systematic way to gather and incorporate feedback from subject matter experts.
• Open-Ended Language Measurement: The use of LLM-as-a-judge or other sophisticated techniques to measure the quality of language outputs.
• Real-World Test Cases: The ability to capture and convert real user traffic into evaluation datasets.
• Customizability: The flexibility to tailor evaluation metrics and processes to specific business requirements, as no out-of-the-box solution is universally effective.

2. A Robust Agent Ops Platform

Agent Ops provides the standardized, end-to-end process for managing the agent lifecycle. A platform built for this purpose needs five key components:

1. Access to LLMs: Seamless connectivity to a variety of large language models.
2. End-to-End Observability: Consistent tracing and monitoring capabilities that work identically across development and production environments.
3. Framework Integration: Compatibility with popular SDKs and authoring frameworks.
4. Open Standards: Built on open protocols like OpenTelemetry to avoid vendor lock-in and ensure interoperability.
5. Security and Governance: Enterprise-grade features like encryption, audit logs, and network controls to protect sensitive data.

Introducing MLflow 3.0

MLflow 3.0 is an upcoming release designed to meet these challenges head-on. It extends MLflow’s established capabilities to provide a comprehensive toolkit for the entire agent lifecycle, built on open standards and integrated with the broader ecosystem.

The key features of MLflow 3.0 include:

• Unified Tracing: With a single line of code, developers get full visibility into their applications. The tracing is built on OpenTelemetry, supports over 20 frameworks, and provides rich visualizations, search capabilities, and integrated assessments.
• Advanced Evaluation: The new MLflow genai-evaluate command simplifies the evaluation process with out-of-the-box metrics while remaining fully customizable with any Python code. It includes UIs for comparing prompt or agent versions side-by-side and tools for sending traces to experts for labeling and review.
• Integrated Tooling: The platform includes built-in capabilities for versioning prompts and agent code, alongside the MLflow Gateway, which provides a secure and unified interface to access various LLM providers like OpenAI and Anthropic.

MLflow on Databricks: Enterprise-Grade Capabilities

While MLflow remains a powerful open-source tool, the managed version on Databricks adds critical enterprise features:

• Enterprise Governance: Full integration with Databricks access controls, permissions, and audit logs.
• Data Intelligence: Incorporates proprietary Databricks research, particularly for advanced LLM-judging and evaluation.
• Lakehouse Integration: All tracing and evaluation data is stored as tables in the Lakehouse, making it directly available for SQL queries, BI dashboards, and other downstream data workflows.

The official release of MLflow 3.0 is scheduled for around July 12th at the Databricks Data + AI Summit.

• Tags:
• LLM
• Evaluation
• Feature
• MLOps
• Observability
• MLflow
• Agent Ops
• Databricks