A hands-on workshop from Langfuse on evaluating AI applications where mistakes are costly. Using a banking assistant as the running example, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, mixing LLM-as-a-judge and code evaluators, and CI gates that block prompt changes when scores drop.
Ishan Anand discusses synthetic personas for market research, drawing parallels to weather forecasting. He outlines three key failure modes—latent confounders, prompt sensitivity, and difficulty predicting actions—and explores techniques like fine-tuning and calibrated prompting to overcome them. Anand stresses the importance of validating personas against human data using distributional metrics and establishing a "noise floor" based on human-to-human consistency, advocating for their use as economic actors and complements to, rather than replacements for, human research.
Agentic AI failures are often predictable system design flaws, not random hallucinations. This summary explores the top three failure modes—infinite loops, hallucinated planning, and unsafe tool use—and provides practical strategies for designing more reliable and robust AI agents.