Kevin Mandia, founder of Mandiant and CEO of Armadin, discusses how AI is revolutionizing cybersecurity. He explains that machine-speed attacks necessitate autonomous defenses, detailing Armadin's approach of continuous AI-driven red teaming to find zero-days and build real-time defense mechanisms. The conversation also explores the transformation of the SOC, the demise of traditional pentesting, and the challenges of building a company at AI speed.
Sam Parsons from PayPal Enterprise Payments explores three distinct ways merchants can accept payments initiated by AI agents: Instant Checkout in ChatGPT (Agentic Commerce Protocol), Google AI Mode (Universal Commerce Protocol), and a custom MCP app with an external checkout page. He details the technical flows, emphasizing tokenization for secure transactions and outlining the trade-offs between in-agent experiences and full merchant control.
Doug Guthrie showcases how Braintrust connects observability with AI agent improvement. Using a support agent, he demonstrates setting up tracing, establishing evaluation baselines, and publishing custom scorers. The workshop delves into Braintrust's Topics feature for discovering unknown failure modes, configuring online automations, and importing real-world traces. A key focus is on automating the improvement cycle: using the Loop assistant for SQL-based trace analysis, integrating coding agents with Braintrust CLI skills to propose code changes, add regressions, run evals, and generate pull requests, thereby closing the feedback loop from production to development.
Charles Frye explains the architecture, performance engineering, and operational challenges of LLM inference engines. He dissects workload types, core components like schedulers and kernel backends, and advanced techniques such as KV caching, CUDA graphs, and speculative decoding, concluding with essential strategies for observability and debugging in production environments.
Browserbase and Microsoft Research unveil their Universal Verifier for computer-use and web agents, addressing the critical flaw of inaccurate LLM-as-judge benchmarks that yield "confident liars." They detail its four guiding principles, demonstrating how it drastically cuts false positives and provides high agreement with human evaluations, leading to better training data. An auto-research experiment highlights the synergy between AI and human expertise in developing robust evaluation systems.
This presentation by Jeff Koss and Leo Platzer from Deasy Labs (now Collibra) tackles the critical challenge of scaling AI chatbots from pilot projects to production with large, messy unstructured datasets. They demonstrate a solution for curating and cleaning 80,000+ SharePoint documents, highlighting the importance of data quality—specifically identifying duplicates, conflicts, sensitive information, and ensuring data freshness—to significantly improve RAG chatbot performance and reduce development time.