Raindrop is building the safety layer for AI agents, detecting and preventing production failures like hallucinations and failed tool calls. The company addresses the paradox that as agents become more capable, they introduce greater complexity and more ways for things to go wrong. Their new Raindrop Simulations product catches critical failures in CI before they ever reach production, democratizing a capability previously limited to large AI labs.
Cedric Clyburn outlines the essential skills for AI engineering, distinguishing it from ML research. He details a three-tier skill stack covering foundational programming (Python, Git, APIs), AI-specific techniques (embeddings, RAG, AI agents), and deployment strategies (containerization, observability). The discussion emphasizes judgment over code generation and highlights key use cases like RAG systems and agent-based solutions.
Abi Aryan discusses the shift from MLOps to LLMOps, framing generative AI deployment as a distributed systems challenge. She highlights the critical role of FinOps due to high model costs, the need for custom tooling over generic agent frameworks, and the importance of a 'decision layer' for agent autonomy. Observability is re-conceptualized as anomaly detection, and she points to dynamic RL environments and strong data engineering as future research priorities and essential skills for AI engineers.
Roberto Milev and Uday Kanagala from Navan discuss the challenges and emerging best practices in building and operating AI agents, drawing parallels with microservices. They highlight the shift to stateful runtimes, innovative memory and context management using 'skills,' and crucial solutions for observability and testing non-deterministic systems. A central theme is the critical need for robust authorization and guardrails, given the blurring line of agentic action, and a preference for mastering single-agent loops before scaling to complex orchestrations. They conclude by outlining the solved and unsolved problems, with cost and debugging remaining significant hurdles.
Uber's Will Bond and Ameya Ketkar present uReview, an in-house automated code review system developed to combat rising review times (3 to 9 hours for first review). They detail why Uber built it over buying—due to Phabricator, agentic SDLC integration, and distributed ownership—and its architecture, including comment deduplication. The presentation highlights their iterative approach using advanced observability (sentiment, addressal rate, agent trajectory) to tune model performance, acknowledging that "the model never knows that it is wrong." They discuss empowering hundreds of teams with custom review agents and skills, sharing impressive results: 25,000 comments weekly, 67% addressal rate, and 60% cost reduction. Finally, they explore the evolving role of human engineers in an agentic SDLC, predicting an "expanded outer loop" focused on architecture and domain expertise rather than direct code review.
Safia Abdalla discusses Warp's cloud agent platform, emphasizing its core principle of absorbing complexity from the user. She details features like flexible sandboxes, multi-harness support, and API-driven agent orchestration. The talk highlights how agents manage Warp's open-source repository—from issue triage to PR review—and introduces the "potter's workshop" analogy as a superior model to the "software factory" for modern, human-centric software development.