Oct 18, 2026

Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

A hands-on workshop from Langfuse on evaluating AI applications where mistakes are costly. Using a banking assistant as the running example, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, mixing LLM-as-a-judge and code evaluators, and CI gates that block prompt changes when scores drop.

Apr 10, 2026

Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

This workshop by Mahmoud Mabrouk, CEO of Agenta AI, delves into building calibrated LLM-as-a-judge evaluations that reliably align with human judgment. It highlights how miscalibrated judges lead to false confidence and presents a practical workflow, including designing use-case specific metrics, detailed data annotation, and optimizing judge prompts using the GAPA algorithm. The talk emphasizes the importance of iterative debugging, model selection, and custom reflection templates for achieving trustworthy and effective LLM evaluations.