Abhi Arya from Reducto shares how the team went from a one-tool-per-endpoint MCP server that confidently built workflows nobody could explain to a design that makes uncertainty visible, asks instead of guessing, and improves itself through instrumentation. The talk centers on one question: who pays when the agent is wrong?
George He, head of engineering at LlamaIndex, explores when agents need pre-indexed vector search and when plain file traversal with grep is enough. He contrasts Claude Code's embedding-free approach on code with the harder reality of large, messy, multimodal, permissioned company documents, and proposes a harness of hybrid retrieval, metadata filtering, grep, and reading tools backed by LlamaParse and LiteParse. The talk closes with a live demo of an agent building a cash-flow table from 135 Alphabet filings, grounded in source files.
This talk by Jess Wang from Braintrust delves into the critical importance of evaluations (evals) in AI development, moving beyond subjective "vibes" to data-driven decision-making. She breaks down the four core components of an eval—dataset, task, scoring, and experiments—and illustrates the continuous improvement "eval flywheel" integrating observability. The central experiment compares agentic search against vector search for a coding agent tasked with finding buggy code in Microsoft's TypeScript Go repo. While both methods achieved identical accuracy, vector search proved four times more expensive due to its inability to capture surrounding context within code chunks, leading to excessive, inefficient searches. Wang concludes with practical caveats and best practices for conducting effective LLM evals.
Vlad Luzin, co-founder and CTO of Band, argues that connecting AI agents is a distributed systems problem that current tools such as MCP, A2A and messaging platforms do not solve. He lays out what is required for real agent collaboration, including ordered real-time transport, persistence, runtime binding, agent-first abstractions and governance, then demonstrates Codex, LangGraph and Claude Code agents finding and working with each other under full human-in-the-loop visibility.
Charlie Holtz, co-founder and CEO of Conductor, shares six principles he has observed among the best AI-assisted builders, including staying near the frontier, creating slop-free zones, and giving agents cloud sandboxes. He also demos real-time collaboration and phone-initiated agents, and argues for orchestras over factories, with humans at the center of AI-driven development.
Ryan Cooke of WorkOS argues that software factories should be judged by customer outcomes rather than PR counts or AI-generated code volume. He describes how WorkOS embedded its product engineering process into its factory through TARS, Horizon, a PM agent that drafts Hilltop docs, and an internal MCP gateway connected to Snowflake, along with bug triage, owned sandboxes, a memory layer and outcome-focused metrics.