Fundamentals of Data Engineering • Matt Housley & Joe Reis • GOTO 2025
Joe Reis and Matt Housley, authors of "Fundamentals of Data Engineering," reflect on the book's principles three years after its publication. They discuss how the rise of AI has created both powerful tools and dangerous "bear traps" for engineers, the critical role of expertise in a world of AI-generated content, and why foundational knowledge is more important than ever.
Three years after publishing “Fundamentals of Data Engineering,” co-authors Joe Reis and Matt Housley reflect on its enduring relevance in a landscape dramatically reshaped by Generative AI. While the book was written “the hard, rugged pioneer way” before the AI explosion, its core principles have become more critical than ever.
The Genesis of a Foundational Text
The book was born from the authors’ consulting experiences in the post-Hadoop, cloud-native era. They observed a chaotic environment where powerful new tools made it easy to build “haphazard junk.” A key motivation was to address the “curse of familiarity,” where engineers misapplied old paradigms to new cloud tools or used new, simplified tools without understanding their underlying mechanics.
They saw a need to define the practice of data engineering beyond a list of tools. At the time, web searches for “what is data engineering” yielded tool-centric, vendor-driven articles. The authors aimed to provide a holistic, foundational view of the data lifecycle, focusing on principles that transcend any specific technology.
“If you were to do a web search on what is data engineering… you come up with a bunch of usually really bad articles… it was very tool-centric and it was very vendor-driven.”
The AI Revolution: A Double-Edged Sword
The rise of AI, particularly LLMs, represents the single biggest change since the book’s release. This has introduced both powerful accelerators and significant risks.
AI as an Assistant vs. a Crutch
The authors draw a distinction between using AI as a “red team” and a “blue team.”
- Red Team (Constructive Analysis): AI can be a powerful partner for analysis and feedback. Joe Reis describes using multiple LLMs as “development editors” to review his writing, offering perspectives he might miss. This aligns with mathematician Terrence Tao’s idea of using AI to find holes in your work, suggest improvements, and challenge your assumptions.
- Blue Team (Mediocre Generation): Conversely, allowing AI to perform the primary work can lead to bland, mediocre output. Tools like the modern version of Grammarly can “shoehorn your writing into a bland way of writing.” For data engineers, this is more dangerous. Relying on an AI assistant to write SQL or code without deep understanding is a “bear trap” for junior engineers who can’t spot subtle but critical errors.
“I get very nervous when I see young data engineers ask ChatGPT or Claude or whatever how to do X and then they get some response back… if you haven’t spent time writing SQL by hand, then how are you going to know when it’s feeding you bad information?”
The Enduring Value of Expertise and Fundamentals
In an age where AI can “conjure” code and content, the role of human expertise is in question. The authors argue it’s more critical than ever. The tech industry’s current trend of gutting junior roles is a “big self-own,” as it prevents the next generation from developing the deep experience necessary to effectively manage AI tools and build robust systems.
True insight doesn’t come from a prompt; it comes from the “tedium” and “grind” of solving problems repeatedly. An expert knows the answer to a common problem because they have seen it and solved it multiple times. AI, by contrast, is creative and stochastic, which isn’t always desirable when a clear, correct answer exists.
The principles in the book remain 95% relevant because they address the timeless challenges of the data lifecycle:
- Choosing the Right Tool: A core problem the authors repeatedly saw was companies using the wrong storage system for their workload—for example, attempting high-throughput transactional updates on a columnar database like Snowflake. This fundamental mismatch leads to poor performance and high costs, a problem AI doesn’t solve.
- The Resurgence of “Classical” Techniques: As companies grapple with the immense cost and hallucinatory nature of LLMs, there’s a renewed appreciation for classical, statistical-based methods. For instance, classical NLP is often more efficient and reliable for the heavy lifting of data processing, with LLMs used more surgically. This creates a demand for data engineers who can scale these traditional techniques.
- Data Quality: The quality of AI models is dependent on the quality of their training data. As Andrew Ng points out, scaling laws are meaningless if you’re scaling on “junk data.” The data engineering principles of data quality management, filtering, and governance are becoming central to building effective AI systems.
Ultimately, AI is an accelerator. It can accelerate good practices, but it can also accelerate the creation of insecure, unethical, and low-quality data systems. Without a solid foundation in the principles of data engineering, one cannot effectively wield these powerful new tools.