Tokenless

Arize

Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.

Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

A financial agent produces confident investment advice with fabricated details, redundant tool calls and bad summaries. This talk shows how traces, evaluations, and a coding agent connected through the Arize CLI and skills can investigate those failures, propose fixes, and test them with experiments. It covers why evaluations matter over vibes-based shipping, how judges and costs fit in, and why human review and application-specific evals are still essential.

Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.