Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

Doug Guthrie, a Solutions Engineer at Braintrust, presented a workshop on mastering AI observability, focusing on how Braintrust’s platform enables the creation of high-quality AI agents by transforming raw trace data into actionable insights.

The Foundation: Tracing and the AI Agent Improvement Flywheel

Guthrie emphasized that tracing is the bedrock of observability, allowing developers to understand every step, input, and output of their agents. Without comprehensive tracing, identifying quality deterioration becomes a black box. Braintrust supports diverse integration methods, accommodating various agent frameworks (like OpenAI Agents SDK), AI providers, and programming languages (Python, TypeScript, Java, Ruby, Go).

The core philosophy presented is the “flywheel” of agent improvement: production data must inform development. This involves a continuous loop of developing changes, testing them via evaluations, deploying to production, observing, and then deriving insights at scale from those observations. The platform aims to move beyond passive observation to active observability, delivering insights directly to developers to facilitate rapid agent improvement.

Generating Actionable Signal from Production Noise

To make sense of the massive data generated by AI agents, Braintrust focuses on two primary signal generation mechanisms:

1. Scores (Evaluators)

Scores provide a measurable way to assess agent output and identify specific failure modes. Guthrie detailed several types:

• Code-based scores: Fast, cheap, and ideal for objective checks like schema validation or tool call order. These can also invoke LLMs for more complex logic.
• LLM judges: Provide subjective, human-like assessment based on defined criteria.
• Human-aligned LLM judges: Crucial for calibrating LLM judges against human review, especially for online scoring, ensuring their reliability.

Best practices for scoring include providing LLM reasoning, using binary scoring for initial harshness, creating difficult test cases, including few-shot examples in prompts, and potentially using stronger models for scoring than for generation. Scores are used in two contexts:

• Offline Evals: Part of CI/CD pipelines to assess improvements or regressions before deployment.
• Online Scoring: Applied to live production traffic to identify patterns in real-time user interactions, flagging issues like low routing accuracy.

2. Topics

Topics are designed to uncover “unknown unknowns”—failure modes or patterns not covered by predefined scores or evaluation datasets. This feature helps understand silent failures, edge cases, and even potential feature requests by clustering patterns in agent interactions. Braintrust offers:

• Built-in facets: Task (user intent), Sentiment (user feelings), and Issues (common agent problems).
• Custom facets: Users can define their own prompts and pipelines to identify patterns relevant to specific business logic or concerns.

The Topics Processing Pipeline

This complex pipeline operates behind the scenes to scale insight generation:

1. Preprocessor: Transforms raw, large traces into meaningful, smaller inputs for the LLM.
2. Facet Application: Passes preprocessed data to chosen facets (built-in or custom).
3. Summarization: Generates a summary of the interaction.
4. Vector Embedding: Creates a vector embedding of the summary.
5. Topic Map Generation: Uses embeddings to create visual topic maps, label traces, and classify them, making filtering and analysis trivial. Braintrust uses its own cost-effective, hosted models for this pipeline, optimized for scale (6 cents/million input tokens, 40 cents/million output tokens).

Workshop: From Tracing to Automated Improvement

The workshop demonstrated a practical implementation using a Python support agent built with the OpenAI Agents SDK:

1. Setup: Participants configured a Braintrust organization, project, AI providers, API keys, and set up the companion GitHub repository with the Braintrust CLI and coding agent skills.
2. Tracing & Baseline Evals: The support agent was instrumented, and an initial set of offline evaluations was run to establish a performance baseline.
3. Online Observability Configuration: Custom scorers were published to Braintrust, and online automations were configured to run these scorers on incoming traces. Custom topic facets were defined (e.g., support workflow issue), and topic processing settings were adjusted for factors like sampling, windowing, and conversation grouping.
4. Data Ingestion & Analysis: A batch of 968 sample support traces was imported, showcasing how Braintrust processes and labels them with topic classifications and scores in real-time. Guthrie demonstrated reading topic maps to identify recurring workflow failures, such as “order record retrieval failure” and “escalate to human tool errors.”

Automating the Improvement Loop

Braintrust facilitates turning observations into agent improvements:

• Loop Assistant & SQL Analysis: Braintrust’s embedded Loop assistant leverages SQL queries against the underlying BrainStore database (Braintrust’s custom-built backend) to analyze trace data. This allows for flexible querying of all trace and span metadata, enabling the identification of representative failures at scale.
• Bringing Failures to Evals: Identified failure cases can be directly added back to evaluation datasets, ensuring that subsequent agent changes are tested against real-world regressions.
• Coding Agent with Braintrust CLI Skills: The workshop demonstrated how a coding agent, equipped with Braintrust CLI skills (e.g., bt sql, bt eval), can automate the entire improvement process: investigating logs, proposing code and scorer changes, adding regression examples to datasets, and running evaluations. This extends into the repository, allowing agents to act on findings directly.
• GitHub Automation: A powerful demonstration showed how this process culminates in an automated GitHub pull request, packaging findings, explanations, and evaluation evidence for human review, significantly accelerating the development cycle.

Key Q&A Insights and Features

Audience questions covered:

• Deployment: Braintrust offers full SaaS, BYOC (Bring Your Own Cloud), and hybrid deployment models, with the data plane residing in the customer’s VPC for hybrid and BYOC.
• Sampling Rates: Higher rates for new agents, scaling back as agents mature. Topics can be a cost-effective alternative to LLM judges for classification tasks at scale.
• Project Scope: Projects serve as containers for relevant resources (logs, evals, scores) for a specific use case or agent, with future plans for more cross-project visibility.
• Access Controls: Insights can be surfaced to reviewers via PRs without granting direct access to sensitive trace data.
• Evolving Evals: Evaluation datasets and scores should evolve with the agent, incorporating production-identified edge cases and failure modes.
• Remote Evals & Playground: A feature allowing engineers to expose evaluations to a playground, letting collaborators (e.g., product managers) adjust prompts and models remotely while execution remains on the evaluation server, streamlining iteration and experimentation.
• Underestimated Features: Guthrie highlighted the Braintrust CLI for its power in automating workflows for coding agents and remote evals for empowering cross-functional collaboration on agent tuning.

Overall, the workshop provided a comprehensive view of how Braintrust integrates tracing, advanced signal generation (scores, topics), and automation to create a robust, active observability platform for continuous AI agent improvement.

• Tags:
• Production AI
• AI Evaluation
• LLM Agents
• Tracing
• AI Observability
• AI Flywheel
• Topic Modeling
• Braintrust
• Agent Development
• AI Monitoring
• LLM Scoring
• Active Observability
• Braintrust CLI
• GitHub Automation
• Agent Improvement