Are Your Tests Slowing You Down? • Trisha Gee • GOTO 2025

Trisha Gee argues that while automated testing is crucial for developer confidence (likening it to “Xanax for programmers” as Kent Beck put it), several issues hinder developer productivity. She identifies three main areas that slow down testing: writing tests, troubleshooting tests, and running tests, offering practical strategies and tools for each.

Writing Tests Takes Time

While social media polls indicate that writing tests is a primary bottleneck, Gee suggests that the thinking behind tests, rather than the typing, consumes the most time. She contrasts traditional IDE features for test generation with AI-powered assistants:

• IDE Tools: These are deterministic and fast, allowing developers to maintain flow. Gee demonstrates creating custom live templates for consistent test structures.
• AI Tools: While theoretically promising, live demos reveal AI assistants can be slow and non-deterministic, often failing to generate code or requiring significant manual correction. She notes AI’s strengths lie in generating test data (a tedious task), considering edge cases not immediately apparent to humans, and acting as a companion rather than a replacement for the developer.
• The “Thinking” Bottleneck: Drawing on Dave Farley’s insight that “the keyboard is not the bottleneck,” Gee emphasizes that designing tests—understanding expected system behavior, edge cases, and failure scenarios—is the most time-consuming part. This cognitive effort is why two people at one keyboard (pairing) can be productive, as it offloads some of this thinking.
• Collaborative Test Design: Working with QA professionals, who excel at breaking systems, can enhance test quality. Developers need to adopt a “stabilizer mindset” (methodical, slower pace) when writing tests, distinct from their “producer/innovator” mindset during coding, to ensure comprehensive coverage.
• Investment for Future Savings: Investing time upfront in writing well-structured tests pays off by significantly reducing future troubleshooting efforts.

Troubleshooting Tests Takes Time

Troubleshooting tests, especially those failing in CI, presents a significant challenge. Gee highlights common pitfalls and solutions:

• Poor Test Design: Unhelpful test names (e.g., testIncrement) and assertions (e.g., assertTrue without context) obscure intent, making failures difficult to diagnose. Tests should clearly state what is expected and why.
• Effective Test Structure: Gee advocates for the given/when/then structure for clarity and assertAll (JUnit 5) to run all assertions and report multiple failures simultaneously.
• Causes of Failure: Failures aren’t always due to incorrect production or test code. They can stem from refactoring, data changes, infrastructure issues (especially in CI), or load/usage. Identifying the root cause helps determine whether it’s a developer’s responsibility or an external system issue.
• Data-Driven Triage: Tools like Develocity’s build tracking and build scans provide crucial data, including infrastructure details and dependencies, to pinpoint when and why a test began failing. Develocity’s AI-powered test failure grouping can categorize multiple failures under common causes.
• Scientific Debugging: The process involves investigating, reproducing locally (using a debugger), making changes, and iteratively running tests. The goal is a fast feedback loop, avoiding the slow, painful cycle of committing, pushing, and waiting for CI results.

Running Tests Takes Time

Long test execution times lead to context switching and frustration. Gee outlines several strategies for accelerating test runs:

• Build Cache: A build cache (e.g., Develocity’s build cache) significantly reduces build and test times by reusing outputs for unchanged inputs. Remote build caches are particularly effective, allowing local builds to leverage results from CI, preventing redundant computations.
• Parallel Execution: Running tests in parallel on local machines (limited by CPU cores) and in CI environments (using multiple agents or test distribution) dramatically reduces execution time. Test distribution tools (like Develocity’s) can massively parallelize tests across many agents.
• Predictive Test Selection (PTS): This technique, originating from Meta’s research, uses machine learning to identify and run only the tests most likely to fail based on code changes. This can reduce the number of executed tests by up to two-thirds, providing a faster sanity check locally while full test suites run on main branches.
• Combined Approach: Leveraging build caches, predictive test selection, test distribution, and observability together creates a fast feedback cycle, crucial for managing large, complex projects or monoliths.

Write Better Tests

Beyond acceleration techniques, the fundamental quality of tests is paramount:

• Engineering Discipline: Test code demands the same software engineering rigor as production code, including separation of concerns, descriptive naming, and modularization. Understanding and deliberately accepting design trade-offs (e.g., high isolation vs. setup/teardown time) is essential.
• Refactor and Delete: Tests should not be immutable. Regularly refactoring and even deleting tests that no longer provide unique value (e.g., those that haven’t failed in years) is vital for maintaining a lean, effective test suite. Identifying such tests automatically is challenging, requiring human judgment and courage.

Conclusion

Automating as much as possible, utilizing tools (like IDEs and DPE platforms) for efficiency, and making a conscious time investment in test design are key. Writing good tests forces critical thinking about system behavior, leading to easier troubleshooting, less flakiness, and, crucially, provides living documentation of the system’s expected functionality.

• Tags:
• Testing
• Developer Productivity Engineering
• DPE
• Software Tests
• Flaky Tests
• Test Automation
• Build Cache
• Predictive Test Selection
• Machine Learning in Testing
• AI in Testing
• Develocity
• Gradle
• IntelliJ IDEA