Advanced Workshop: Mastering AI Observability — Doug Guthrie, Braintrust

Doug Guthrie showcases how Braintrust connects observability with AI agent improvement. Using a support agent, he demonstrates setting up tracing, establishing evaluation baselines, and publishing custom scorers. The workshop delves into Braintrust's Topics feature for discovering unknown failure modes, configuring online automations, and importing real-world traces. A key focus is on automating the improvement cycle: using the Loop assistant for SQL-based trace analysis, integrating coding agents with Braintrust CLI skills to propose code changes, add regressions, run evals, and generate pull requests, thereby closing the feedback loop from production to development.