Gareth Yoder of Vanguard describes modernizing a cash ledger handling up to 30 million calls per day by running data, producer, and consumer migrations in parallel. Key themes include exposing legacy code to AI through an MCP server, reducing a seven-day research cycle to fifteen minutes, reimagining rather than lifting and shifting, and maintaining observability throughout the migration.
Rachna Srivastava discusses building a legally defensible AI system for financial fraud detection, emphasizing that generative AI has eroded traditional trust. Her team at DFPI engineered an offline, hardware-secured data pipeline using Kafka, Spark, and semantic routing to ensure explainability, reproducibility, and auditability. They implemented a one-way data diode for secure learning and Apache Iceberg for time-travel queries, asserting that "trust is a physical property" built into the system's core.
Ekaterina Gorshkova, author of "Kafka for Architects", discusses her journey with Apache Kafka, from its early adoption in enterprise integration to its modern role as a foundational layer for AI and ML systems. The conversation covers key architectural decisions, the limitations of Kafka, and why the real challenges are often organizational rather than technical.
Félix GV, architect of LinkedIn's Venice database, explains its unbundled, planetary-scale architecture. He covers their rigorous chaos engineering practices, the trade-offs between consistency and availability in a derived data system, and an experimental integration with DuckDB for advanced analytics and data exploration.
Jacqueline Cheong and Robin Tang, founders of real-time data streaming platform Artie, discuss their journey from identifying the critical need for fresh data at companies like OpenDoor to building a production-ready solution, acquiring their first major customer Substack via a cold email, and navigating the complex technical challenges of real-time data processing at scale.