Playground in Prod - Optimising Agents in Production Environments — Samuel Colvin, Pydantic

The central challenge in deploying AI agents isn’t just the initial launch, but the continuous improvement and optimization of their performance in a live environment. This involves updating prompts, models, and other parameters without disruptive redeployments. The process can be broken down into three key phases: evaluating performance, autonomously optimizing prompts, and deploying those improvements dynamically.

Core Concepts: GEPA and Managed Variables

Two primary tools form the foundation of this workflow:

• GEPA (Genetic Pareto): An optimization library that functions as a genetic algorithm. It iteratively “breeds” the best-performing prompts to find superior versions. It operates on a string, which can be a simple text prompt or a JSON object representing a more complex configuration. GEPA takes candidates from the “Pareto frontier”—the set of best-known solutions—to generate and test new candidates, effectively hill-climbing towards an optimal configuration.
• Logfire Managed Variables: This feature extends traditional prompt management. Instead of just managing text, it allows any object definable with a Pydantic model (e.g., a configuration including the model name, temperature, and instructions) to be managed and updated live from the Logfire platform. This enables developers to change an agent’s behavior in production without restarting the service, facilitating A/B testing and rapid iteration.

The Use Case: Analyzing Political Ancestry

To demonstrate these concepts, a practical task was chosen: analyzing the Wikipedia pages of UK Members of Parliament (MPs) to identify their political ancestors.

The core of the agent is a structured output task using Pydantic AI. The agent is given the HTML content of a Wikipedia page and is tasked with extracting a list of PoliticalRelation objects.

A key challenge highlighted is the difficulty in getting LLMs to strictly adhere to the constraint of identifying only ancestors (parents, grandparents, etc.) while ignoring other relations like spouses or children, which they tend to include by default.

Phase 1: Evaluating Performance with Logfire Evals

Before optimization, it’s crucial to establish a baseline and a method for measuring performance.

1. Golden Dataset: A “golden” dataset is created, containing the manually verified correct political ancestors for a subset of MPs. This serves as the ground truth for evaluation.
2. Running Evals: The logfire library is used to run evaluations. The agent is run on each case in the test set, and its output is compared against the golden dataset.
3. Comparing Variants: The Logfire UI allows for direct comparison between different experimental runs. For instance, a basic, one-line prompt can be compared against a more detailed “expert” prompt. The comparison view shows aggregate metrics (like accuracy) and, crucially, allows drilling down into individual cases where the performance differs. This helps identify specific failure modes, such as incorrectly identifying a public figure (e.g., a sports commentator) as a political relation.

# Command to run a comparison between two different prompts
uv run main.py compare --split test --prompts initial expert

Phase 2: Autonomous Prompt Optimization with GEPA

With a reliable evaluation process in place, GEPA can be used to automate the discovery of a better prompt.

• The Proposer Agent: The optimization process itself uses an LLM. A “proposer agent” (in this case, another Pydantic AI agent) is tasked with generating new system prompts. It receives context from GEPA about previously successful prompts and their performance scores.
• The Optimization Loop:
  1. GEPA’s proposer agent generates a new candidate prompt.
  2. This new prompt is evaluated against the test dataset using the Logfire eval function.
  3. The performance score is fed back to GEPA.
  4. GEPA uses this feedback to inform the next generation of prompts, selecting and combining elements from the best performers.
• Result: After running the optimization for a set number of calls, GEPA produced a significantly more verbose and detailed prompt. This new prompt explicitly listed rules and examples for what constitutes a political ancestor, achieving a higher accuracy score (~96.7%) compared to the manually written “expert” prompt (~92%).

Phase 3: Deploying Improvements with Managed Variables

Once an optimal prompt is found, it needs to be deployed to the live application.

1. Live Application: A simple FastAPI server demonstrates a live agent that can answer questions about MPs.

2. Defining a Variable: A Pydantic model defines the configuration that can be changed dynamically. This configuration is wrapped in a logfire.variable() call within the application code.

   class AgentConfig(BaseModel):
       instructions: str
       model: str
       max_tokens: int
   
   # In the application code
   config_variable = logfire.variable(
       "mp-search-config",
       default=AgentConfig(...)
   )
   

3. Updating in Production:

   • The new, GEPA-optimized prompt is pasted into the instructions field of the managed variable in the Logfire UI.
   • The targeting rules are updated to direct 100% of traffic to this new version.
   • The live FastAPI application automatically fetches the updated configuration on its next invocation, immediately changing its behavior to use the new prompt. This happens instantly, with no code changes or service restarts.

This workflow creates a powerful feedback loop: production data and feedback signals inform evaluations, which power autonomous optimization with GEPA, and the resulting improvements are deployed back into production seamlessly using managed variables.

• Tags:
• Prompt Engineering
• Feature
• MLOps
• Logfire
• Pydantic
• GEPA
• LLM Evals
• Agent Optimization