An evaluation platform is more than a simple test runner; it's a complex system for creating shared definitions of quality. This talk explores the evolution of eval platforms from basic spreadsheets to sophisticated, integrated systems, highlighting the hidden data and systems engineering challenges involved in making them credible, scalable, and usable for building trustworthy AI agents.
Explore the lifecycle of building and operating a robust serverless platform, treating it as a product for internal and external consumers. This talk covers the crucial phases of ideation, construction (using the AWS Well-Architected Framework), and operation, emphasizing a consumer-centric mindset, operational excellence, and strategies for long-term success.
Ekaterina Gorshkova, author of "Kafka for Architects", discusses her journey with Apache Kafka, from its early adoption in enterprise integration to its modern role as a foundational layer for AI and ML systems. The conversation covers key architectural decisions, the limitations of Kafka, and why the real challenges are often organizational rather than technical.
Mihail Eric, Head of AI at Monaco and Stanford Lecturer, discusses the fundamental shift in software engineering due to AI. He explores how the developer's role is evolving from a line-by-line coder to a systems architect and agent director, emphasizing the growing importance of upfront planning, robust validation, and clear articulation to effectively manage AI-driven workflows.
Learn practical strategies to adapt third-party MCP server tools for production AI applications. This talk covers five key practices: curating tools, enhancing descriptions, implementing deterministic guardrails, composing new tools from existing ones, and leveraging tools as simple functions, all demonstrated through a real-world "Spec Reviewer" example.
Jay Wengrow, author of “A Common-Sense Guide to AI Engineering,” breaks down how AI agents work, describing the 'clever hack' of intercepting LLM output to trigger functions. The discussion covers multi-agent architectures for complex tasks, implementing guardrails with regex and judge LLMs, and a pragmatic take on when to use frameworks versus building from scratch. Wengrow emphasizes understanding fundamentals over specific tools to create robust, production-ready AI applications.