Maggie Konstanty, AI Product Manager at Prosus, provides a candid look into the realities of LLM evaluation in production. She argues that standard metrics like accuracy are misleading and advocates for a culture of continuous, goal-oriented evaluation focused on deep failure analysis and understanding real user behavior, asserting that mature teams inevitably build custom tooling to meet their specific needs.
Zach Lloyd, CEO of Warp, explains why the future of software development is moving from local, interactive agents to cloud-native, collaborative systems. He discusses the flaws in the "dev box" sandbox model, the decline of traditional SaaS interfaces in favor of "just-in-time apps," and how platforms like Warp's Oz are providing the necessary orchestration, observability, and access control for teams to effectively deploy AI agents at scale.
AI agents represent a paradigm shift from conversational AI to autonomous systems that can perform actions. This is achieved through an 'agentic loop' combining Large Language Models (LLMs) with tools, as exemplified by the OpenClaw framework, which enables complex, automated workflows while also raising important security considerations.
Maggie Appleton from GitHub Next argues that current agentic tools are flawed by focusing on individual productivity, ignoring the collaborative nature of software development. She introduces ACE (Agent Collaboration Environment), a multiplayer platform designed to solve team alignment issues by integrating planning, development, and shared context in a real-time, sandboxed environment.
This talk explores how to overcome the context window limitations that prevent AI agents from accessing large APIs. It introduces "Codemode," a technique where agents write code against a typed SDK in a secure, sandboxed environment, moving beyond static tool definitions and enabling full API accessibility.
As the number of AI agents grows, humans become the bottleneck. This talk introduces AgentCraft, a game-inspired orchestrator that draws lessons from Real-Time Strategy (RTS) games to enhance human-agent collaboration, improve visibility, and increase agent autonomy, ultimately raising the ceiling of what's possible.