Oct 23, 2026

Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

Stephanie Jarmak, who moved from astronomy into building coding benchmarks and agent evals, argues that most agent evaluations are easy to misread. She covers how harness choice can swamp model differences, why public benchmarks serve shortlisting rather than decisions, how to design experiment-style evals with realistic tasks, repeated runs, and full traces, and how to set quality gates, manage human review load, verify agent records, and treat production as the final test.

Oct 20, 2026

Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

Al Chen explains what it takes to get AI observability software through the security and compliance reviews of major banks. He covers build pipelines, agent-driven CVE remediation with verifiable exploit tests, ClickHouse backup and restore verification, PII and MNPI leakage through LLM judges, and the tradeoffs of zero leakage and data sovereignty, with an emphasis on showing evidence to shorten time to production.

Oct 19, 2026

How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase

Evan Kormos of Coinbase presents a practical framework for bringing AI agents into policy-driven, regulated workflows. He models processes as DAGs with loops inside each node, argues that a loop is only as trustworthy as its check, and shows how evals and traces create a new confidence mechanism. Agents then move through supervised, sampled, and exception-only stages.

Oct 11, 2026

Let Your Agent Cook: Using Skills to Evaluate and Improve Your App — Ankur Duggal, Arize AI

A financial agent produces confident investment advice with fabricated details, redundant tool calls and bad summaries. This talk shows how traces, evaluations, and a coding agent connected through the Arize CLI and skills can investigate those failures, propose fixes, and test them with experiments. It covers why evaluations matter over vibes-based shipping, how judges and costs fit in, and why human review and application-specific evals are still essential.

Oct 11, 2026

Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI

Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.

Oct 10, 2026

Small Models, Big Results: Training a Finance Agent for Under $500 — Charles Dickens, Snorkel AI

Snorkel AI and UC Berkeley's Sky Computing Lab show that a 4B Qwen3 model, trained with reinforcement learning for under $500 using the rLLM framework, beats its 235B sibling on expert-validated financial question answering built from SEC 10-K filings. The talk covers the FinQA dataset construction and verification, common failure modes of frontier models, transfer to multi-table and general tool-use tasks, and ablations showing that simple data and binary rewards outperform more elaborate setups, since the real bottleneck is disciplined tool use rather than reasoning depth.