Advancing the Cost-Quality Frontier in Agentic AI // Krista Opsahl-Ong // Agents in Production 2025

Enterprises are increasingly looking to apply AI agents to high-impact problems, but moving from a prototype to a production-ready system presents significant challenges in cost, quality, and manual effort. The goal is to push the “cost-quality frontier”—achieving higher quality at a lower cost—which requires leveraging the latest research in agent optimization.

Common Enterprise Use Cases for AI Agents

Several patterns of agentic AI use have emerged across enterprises:

• Document Understanding: This involves automating the extraction of structured information from unstructured documents. A common example is processing tens of thousands of monthly invoices to extract key data for entry into SAP, or parsing millions of product spec PDFs on an e-commerce platform to surface relevant information to customers.
• Knowledge Assistance: These agents act as expert assistants, powered by internal documentation. Use cases range from internal-facing deep research tools for analysts to external-facing chatbots that must adhere to specific brand guidelines and tones.
• Custom Solutions & Chaining: Enterprises often need to fine-tune a custom LLM for a specific task, like generating article titles. Furthermore, there is a growing need to chain these different components and agents together to create more complex, multi-step workflows.

The Triad of Productionization Challenges

Getting these agents into production is difficult due to three main obstacles:

1. Extensive Manual Effort: Building a performant agent involves numerous design decisions—choosing the right models, crafting effective prompts, selecting appropriate tools, and defining the overall architecture. This process often devolves into a lengthy cycle of manual setup and trial-and-error.
2. The Cost-Quality Trade-off: There is an inherent trade-off between the performance of an agent and its inference cost. High-quality models are typically expensive, which is often unfeasible for high-volume tasks like information extraction. Pushing the cost-quality curve (up and to the left on a graph plotting quality vs. cost) requires significant investment in research and optimization.
3. Ambiguous Evaluations: Measuring the quality of an agent’s output is a major challenge. Tasks can be ambiguous, and labeled data is often unavailable. Without a reliable way to measure quality, it’s impossible to know if an agent is production-ready or to optimize it effectively.

Agent Bricks: A Platform for Optimization

To address these challenges, Databricks developed Agent Bricks, a platform designed to help enterprises easily create and optimize AI agents for both cost and quality. It provides “bricks” for common use cases like Information Extraction, Knowledge Assistant, and a Multi-agent Supervisor for chaining tools.

The workflow is designed to be straightforward:

1. High-Level Specification: The user starts by selecting a task (e.g., Knowledge Assistant) and providing a simple, natural language description of the desired agent, along with any relevant data sources like internal documents.
2. Automated Evaluation: Agent Bricks automatically generates a suite of evaluations. It uses custom LLM judges to score agent responses on critical metrics such as completeness, groundedness, relevance, and safety, aligning the evaluation with the user’s expectations.
3. One-Click Optimization: With a single “Optimize” button, the platform initiates a comprehensive optimization process. It systematically sweeps through numerous methods and configurations to find the optimal balance of cost and quality for the specific task.

Behind the Optimization Engine

The optimization step leverages a variety of advanced, research-driven techniques, including:

• Automatic Prompt Optimization: Instead of manual prompt engineering, the system systematically proposes and evaluates a wide range of prompts to find the best-performing ones.
• Reinforcement Learning (RL): It employs label-free RL methods like Tao, a recent development from the Databricks research team, to continuously adapt and improve the agent over time based on new queries, even without explicit labels.
• Systematic Tuning: The platform also optimizes tool choices, model selections, and fine-tuning parameters.
• DSPy Framework: The principles and components of the DSPy framework are integrated into the platform to structure and optimize the agent programs.

The results of this process are significant. On an information extraction benchmark, optimization with Agent Bricks was shown to achieve the same level of quality for approximately 1/10th of the cost, or a 20% increase in quality for the same cost on cheaper models.

• Tags:
• AI Agents
• Prompt Engineering
• LLM
• Feature
• MLOps
• Databricks
• DSPy
• Cost Optimization