Evaluation

Reading Group July 2026 - Loop Engineering

Reading Group July 2026 - Loop Engineering

This session provides an in-depth exploration of Loop Engineering, a paradigm shift from manual AI prompting to designing autonomous systems that orchestrate AI agents. Speakers share practical experiences, from building production-grade platforms with automated code generation and adversarial AI reviews to experimental loop structures and foundational infrastructure layers. Key discussions address challenges like managing token costs, preventing agent chaos, and implementing robust verification mechanisms for industrializing software development.

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

Nubank, serving 135 million customers, uses AI agents for support. The talk reveals how simulated data for evaluations (evals) has enabled them to ship AI agents 20x faster. By addressing the bottleneck of multi-turn, stateful eval data, Snowglobe's grounded simulations create realistic customer interactions, allowing rapid testing, derisking, and significant improvements in customer satisfaction and self-service rates, even for open-source model experimentation.

Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains — Brendan Rappazzo

Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains — Brendan Rappazzo

Morgan Stanley's AlphaLab is an open-sourced multi-agent system designed to automate quantitative research. Initially, AlphaLab 1.0 automated code generation, backtesting, and experimentation. Facing challenges, AlphaLab 2.0 evolves to prioritize building robust, verifiable environments, which serve as reinforcement learning signals, enabling the system to meta-optimize itself. This shift redefines the human role from performing research to designing these critical environments.

The Next Frontier of AI Is Spatial Intelligence | Fei-Fei Li on a16z

The Next Frontier of AI Is Spatial Intelligence | Fei-Fei Li on a16z

Fei-Fei Li and Yunzhu Li discuss World Labs' acquisition of SceniX, focusing on building "spatial intelligence" and "large world models" to enable robots to understand and interact with the physical world. They elaborate on SceniX's "real-to-sim-to-real" pipeline, emphasizing how simulation, coupled with generative models like Marble, addresses the data bottleneck in robotics by providing consistent, scalable, and efficient training and evaluation environments. The conversation covers the role of counterfactual reasoning, the development of robotics foundation models, and the strategic focus on semi-structured environments for pragmatic, reliable robot deployment.

Recursive Model Improvement — Lee Robinson, Cursor, SpaceXAI

Recursive Model Improvement — Lee Robinson, Cursor, SpaceXAI

Lee Robinson of Cursor outlines a comprehensive strategy for recursive AI model improvement, centered on a two-loop training framework. He details how Cursor enhances both user-feedback-driven outer loops and high-quality evaluation inner loops, introducing novel methods like textual feedback and addressing reward hacking. The discussion extends to scaling compute infrastructure through partnerships with SpaceX, Colossus, and Terafab, and leveraging agent-based automation to streamline research and foster a future where models continuously train and improve themselves.

Build Agents That Run for Hours (Without Losing the Plot) — Ash Prabaker & Andrew Wilson, Anthropic

Build Agents That Run for Hours (Without Losing the Plot) — Ash Prabaker & Andrew Wilson, Anthropic

Explore advanced techniques for building long-running AI agents, moving beyond simple loops. Learn why self-evaluation fails and adversarial evaluators succeed, how to manage context with structured handoffs instead of just compaction, and how to use negotiated 'sprint contracts' and detailed rubrics to build and test complex, full-stack applications autonomously.