Oct 22, 2026

How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

Nathanael Weill of Air Canada explains how to evaluate AI agents when no public benchmark exists and expert time is scarce: start from a known answer, generate the input from it, and check whether the agent recovers it. He covers lessons from a telecom RAG project that moved from one expert-graded test a week to dozens a day, and five ways to build test sets for a supplier contract review agent, along with why code-based checks are preferred over LLM judges.

Oct 13, 2026

Automating document work with Long-Horizon AI Agents at LlamaIndex

Jerry, CEO of LlamaIndex, argues that as AI agents move from answering questions to running long-lived workflows, documents become their most important context source. He walks through the evolution from simple RAG to agent harnesses, explains why PDFs are hard to parse, and presents the company's approach to OCR, extraction with grounding, and document search. He closes by outlining unsolved problems: agent-native document formats, in-place editing, versioning, distilling frontier workflows into cheaper models, and long-horizon background agents.

Oct 11, 2026

RAG Needs a Map: Using GraphRAG to Retrieve Connected Context — Nyah Macklin, Neo4j

A hands-on GraphRAG workshop from Neo4j showing how combining vector search with a knowledge graph gives an agent connected context, multi-hop traversal, and explainable answers. It covers the difference between semantic similarity and relationships, LLM-driven entity and relationship extraction under a fixed schema, building an Aura database and Python pipeline, and configuring an Aura agent with similarity search and text-to-Cypher, with a closing check that shows why visible query evidence matters.

Oct 05, 2026

From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

This presentation by Jeff Koss and Leo Platzer from Deasy Labs (now Collibra) tackles the critical challenge of scaling AI chatbots from pilot projects to production with large, messy unstructured datasets. They demonstrate a solution for curating and cleaning 80,000+ SharePoint documents, highlighting the importance of data quality—specifically identifying duplicates, conflicts, sensitive information, and ensuring data freshness—to significantly improve RAG chatbot performance and reduce development time.

Oct 05, 2026

Evals in AI: A Deep Dive — Tejas Kumar, IBM

This deep dive into AI evaluation (evals) starts by clarifying their role in providing ahead-of-time reliability and security for non-deterministic systems, contrasted with just-in-time harnesses. Tejas Kumar defines evals as "fuzzy unit tests" and outlines their core components: test cases, expected behavior, a judge, and aggregation. He explores various evaluation techniques, emphasizing the challenges and biases of using LLMs as judges, such as position bias, sycophancy, self-preference, and verbosity. The presentation details a practical methodology for calibrating LLM judges against human verdicts using a "golden dataset" to achieve an 80-85% agreement threshold, which can then be enforced as a CI/CD gate. A live coding demonstration illustrates the evolution from simple string matching to a policy-aware LLM judge, showcasing how refining policy and incorporating RAG (Retrieval Augmented Generation) via OpenRAG dramatically improves decision-making. The discussion also covers strategies for maintaining living datasets in production and managing evaluation costs.

Sep 18, 2026

Total Recall: Agent Memory and Harness Engineering — Ignacio Martinez, Oracle

Ignacio Martinez's workshop delves into agent harness engineering, emphasizing that while LLMs are a 'frozen' reasoning core, the surrounding harness is where control and reliability are established. He outlines a seven-layer architecture for building robust agent harnesses, covering critical aspects like hybrid storage solutions, advanced memory engineering (including in-database embeddings and context card management), the semantic layer (Umwelt), agent loops, and context engineering. The session also explores continual learning through 'skill promotion' and addresses practical challenges like managing vast tool sets and optimizing model interactions.