Solving the Hard Problems • Dave Farley • GOTO 2024
Dave Farley argues that modern software development should be treated as an engineering discipline, emphasizing an evolutionary and iterative approach to architecture and design. He outlines key principles for tackling complexity, focusing on optimizing for learning and ease of change through techniques like working in small steps, prioritizing testability, and managing coupling.
The Case for an Engineering Discipline in Software
Modern software development should be treated as a true engineering discipline, which is fundamentally about the application of scientific-style reasoning to solve complex problems. Engineering is not a rigid, cookie-cutter process that leads to a single correct solution; rather, it’s a framework for avoiding mistakes and ruling out bad ideas, leaving a spectrum of good ideas to choose from.
Complex systems, like SpaceX’s Starship, are not born from a single moment of genius. They are the product of exploration, discovery, iteration, and learning. The core tools of our trade are not languages or frameworks, but the skills of problem-solving and, crucially, incremental design. We must organize our work to start projects without knowing the final answer, allowing us to learn and build our systems incrementally.
The “Power Tools” of Software Engineering
The essential characteristics of high-quality software development can be grouped into two main categories: optimizing for learning and managing complexity.
Optimizing for Learning
To build systems that can evolve, we must prioritize our ability to learn and adapt.
- Iteration: The foundational process of making small changes, evaluating the outcome, and refining the system over time. This is how all engineering disciplines make progress, from early aviation to modern software.
- Feedback: Gathering data to understand the impact of our changes.
- Incrementalism: Building the system piece by piece, ensuring each addition rests on a stable, working foundation.
- Experimentation: Treating each change as a small experiment to test a hypothesis.
- Empiricism: Measuring the real-world behavior of our systems in production to guide future development.
This approach mirrors how deep neural networks learn, using a “fitness function” to determine if a change moves the system closer to its goal. In software, our tests serve as this fitness function, validating that each small step is moving in the right direction.
Managing Complexity
As we build, we must actively manage the system’s inherent complexity to maintain our ability to change it. The primary measure of software quality is the ease with which it can be changed.
- Modularity & Cohesion: Grouping related code together so that changes are localized and their context is easy to understand.
- Separation of Concerns: Dividing the system so that each part is focused on a single, well-defined responsibility.
- Abstraction: Hiding the implementation details of a component behind a well-defined interface, allowing other parts of the system to interact with it without needing to know how it works.
- Managing Coupling: Minimizing the dependencies between different parts of the system so that a change in one area does not have unintended consequences in another.
Practical Techniques for Evolutionary Architecture
Architecture as an Evolving Snapshot
Software architecture is not a static, upfront blueprint written in stone. It is a living entity that evolves with the system. It’s best thought of as a “tourist map”—a guide that is topologically correct and helps you navigate, but is not perfectly to scale or rigidly prescriptive. It provides a shared understanding that allows team members to reason about the system and make coherent decisions. This map must be maintained and updated as the system changes. If an architecture cannot change, it is already dead.
Key Practices for Implementation
-
Work in Small Steps: Making tiny, incremental changes is profoundly important. It reduces risk, shortens feedback loops, increases confidence, and lowers the cost of being wrong. This allows for rapid learning and a more economically and technically efficient development process.
-
Prioritize Testability: Testability is central to the engineering approach. By defining desired outcomes as testable predictions (or specifications), we create the fitness function needed for iteration. These specifications should describe what the system should do from a user’s perspective, not how it should be implemented. This applies to all requirements, including performance, security, and resilience, which are critical business needs, not “non-functional” afterthoughts.
-
Prioritize Ease of Change: Continuous Delivery is the ultimate tool for enabling ease of change. The practice of building deployment pipelines optimizes for fast feedback and creates an experimental platform. This naturally encourages the development of systems that are modular, cohesive, and loosely coupled, as these are the easiest systems to test and deploy continuously.
-
Adapt to the Current Context: Avoid designing and building for a hypothetical future. Solve the problem you have right now, but do so in a way that leaves the door open for future changes. This is the essence of “You Ain’t Gonna Need It” (YAGNI). Start with a simple implementation and allow the architecture to evolve as new requirements emerge, just as companies like Amazon evolved from a simple PHP application to a global microservices architecture.
-
Use Platforms as an Abstraction: Platform engineering is a modern approach to good design. Its goal is to reduce the cognitive load on feature-development teams by abstracting away common, complex concerns like security, scalability, and infrastructure. A good platform provides these capabilities in a way that allows teams to focus on delivering business value.
-
Manage Coupling: Strive to minimize and simplify the interactions between components. A useful heuristic is to design the “conversations” between services or modules to be recognizable at a business level. A domain expert should be able to understand the flow of messages in the system, indicating a sensible level of abstraction and well-managed coupling.
Ultimately, software development is an act of exploration and discovery. Adopting an engineering mindset—focused on iteration, feedback, and evolutionary design—is the most effective way to navigate this journey and build great, complex systems.