Oct 05, 2026

Evals in AI: A Deep Dive — Tejas Kumar, IBM

This deep dive into AI evaluation (evals) starts by clarifying their role in providing ahead-of-time reliability and security for non-deterministic systems, contrasted with just-in-time harnesses. Tejas Kumar defines evals as "fuzzy unit tests" and outlines their core components: test cases, expected behavior, a judge, and aggregation. He explores various evaluation techniques, emphasizing the challenges and biases of using LLMs as judges, such as position bias, sycophancy, self-preference, and verbosity. The presentation details a practical methodology for calibrating LLM judges against human verdicts using a "golden dataset" to achieve an 80-85% agreement threshold, which can then be enforced as a CI/CD gate. A live coding demonstration illustrates the evolution from simple string matching to a policy-aware LLM judge, showcasing how refining policy and incorporating RAG (Retrieval Augmented Generation) via OpenRAG dramatically improves decision-making. The discussion also covers strategies for maintaining living datasets in production and managing evaluation costs.

Aug 17, 2026

Reading Group July 2026 - Loop Engineering

This session provides an in-depth exploration of Loop Engineering, a paradigm shift from manual AI prompting to designing autonomous systems that orchestrate AI agents. Speakers share practical experiences, from building production-grade platforms with automated code generation and adversarial AI reviews to experimental loop structures and foundational infrastructure layers. Key discussions address challenges like managing token costs, preventing agent chaos, and implementing robust verification mechanisms for industrializing software development.

Jul 29, 2026

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

Nubank, serving 135 million customers, uses AI agents for support. The talk reveals how simulated data for evaluations (evals) has enabled them to ship AI agents 20x faster. By addressing the bottleneck of multi-turn, stateful eval data, Snowglobe's grounded simulations create realistic customer interactions, allowing rapid testing, derisking, and significant improvements in customer satisfaction and self-service rates, even for open-source model experimentation.

Jul 29, 2026

Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains — Brendan Rappazzo

Morgan Stanley's AlphaLab is an open-sourced multi-agent system designed to automate quantitative research. Initially, AlphaLab 1.0 automated code generation, backtesting, and experimentation. Facing challenges, AlphaLab 2.0 evolves to prioritize building robust, verifiable environments, which serve as reinforcement learning signals, enabling the system to meta-optimize itself. This shift redefines the human role from performing research to designing these critical environments.

Jul 28, 2026

The Next Frontier of AI Is Spatial Intelligence | Fei-Fei Li on a16z

Fei-Fei Li and Yunzhu Li discuss World Labs' acquisition of SceniX, focusing on building "spatial intelligence" and "large world models" to enable robots to understand and interact with the physical world. They elaborate on SceniX's "real-to-sim-to-real" pipeline, emphasizing how simulation, coupled with generative models like Marble, addresses the data bottleneck in robotics by providing consistent, scalable, and efficient training and evaluation environments. The conversation covers the role of counterfactual reasoning, the development of robotics foundation models, and the strategic focus on semi-structured environments for pragmatic, reliable robot deployment.

Jul 15, 2026

Recursive Model Improvement — Lee Robinson, Cursor, SpaceXAI

Lee Robinson of Cursor outlines a comprehensive strategy for recursive AI model improvement, centered on a two-loop training framework. He details how Cursor enhances both user-feedback-driven outer loops and high-quality evaluation inner loops, introducing novel methods like textual feedback and addressing reward hacking. The discussion extends to scaling compute infrastructure through partnerships with SpaceX, Colossus, and Terafab, and leveraging agent-based automation to streamline research and foster a future where models continuously train and improve themselves.