Oct 20, 2026

Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

Al Chen explains what it takes to get AI observability software through the security and compliance reviews of major banks. He covers build pipelines, agent-driven CVE remediation with verifiable exploit tests, ClickHouse backup and restore verification, PII and MNPI leakage through LLM judges, and the tradeoffs of zero leakage and data sovereignty, with an emphasis on showing evidence to shorten time to production.

Oct 19, 2026

The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral

Pamela Fox runs a guided workshop comparing GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 on Microsoft Foundry, showing that benchmarks cannot predict how a model will behave in a specific application. The talk covers counting, RAG groundedness, vision, multilingual failures, tool calling, parallel tool calls in agents, string-check and LLM-as-a-judge evals, Microsoft's ASSERT tool for generating test sets, and DSPy prompt optimization that lets a cheap model satisfy a spec.

Oct 18, 2026

Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

A hands-on workshop from Langfuse on evaluating AI applications where mistakes are costly. Using a banking assistant as the running example, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, mixing LLM-as-a-judge and code evaluators, and CI gates that block prompt changes when scores drop.

Oct 11, 2026

Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

A financial agent produces confident investment advice with fabricated details, redundant tool calls and bad summaries. This talk shows how traces, evaluations, and a coding agent connected through the Arize CLI and skills can investigate those failures, propose fixes, and test them with experiments. It covers why evaluations matter over vibes-based shipping, how judges and costs fit in, and why human review and application-specific evals are still essential.

Oct 11, 2026

Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.

Oct 08, 2026

Beyond the Benchmark: How Parallel Web Systems Builds Reliable Web Agents

Parallel Web Systems engineer explains how the company moved beyond academic web-research benchmarks to a fully automated evaluation flywheel that samples production questions, generates reference answers with an oracle agent, judges with tool-equipped agents, and masks components to diagnose failures, along with the open challenges of evaluating subjective and infrastructure-dependent quality.