CI/CD Evolution: From Pipelines to AI-Powered DevOps • Olaf Molenveld & Julian Wood • GOTO 2025
Olaf Molenveld (CircleCI) and Julian Wood (AWS) discuss the evolution of CI/CD practices. They draw parallels between managing production code and the 'factory' that produces it, covering optimization strategies, local vs. remote development, the rise of platform engineering, and how AI is reshaping DevOps by acting as both an expert system and a guarded collaborator.
This discussion explores the evolution of CI/CD, highlighting that delivering software to users is as crucial as writing the code itself. The core idea is that without a ‘factory’ (the CI/CD pipeline) to produce and deliver the software ‘car,’ the design is useless. This factory creates the essential feedback loop from users that drives product improvement.
The Growing Complexity of the Software Factory
The shift from monolithic architectures to microservices has fundamentally changed software delivery. This transition introduced a web of external dependencies, making it difficult to manage updates and determine which tests to run for specific changes. Consequently, the CI/CD pipeline has evolved from a simple script into a complex ‘meta-application’ or ecosystem that requires its own management and maintenance. Just as a physical factory’s success depends on its machinery, a software project’s success depends on the health and efficiency of its delivery pipelines.
Modern Pipeline Architecture and Tooling
Modern pipeline architectures have adapted to this complexity:
- Evolving Triggers: Pipelines are no longer solely triggered by Git commits. Any event, such as a Docker image update or a new ML model, can initiate a pipeline, creating a chain of events similar to event-sourcing patterns.
- Hybrid Pipeline Definitions: The industry moved from imperative scripts (e.g., Bash, Groovy) to declarative, DSL-based pipelines (YAML) for better readability. Now, to manage rising complexity, there’s a return to ‘CI as code’ using tools like Dagger or TypeScript SDKs, creating a hybrid approach where a CI/CD tool orchestrates logic defined in code.
- The Orchestrator’s Role: Tools like CircleCI act as a central orchestrator, providing a ‘single pane of glass’ to manage everything from simple builds to complex, multi-environment enterprise deployments. They handle both short-running ‘inner loops’ for rapid feedback and long-running ‘outer loops’ that may take hours or days for release validation.
Developer Workflows: Local vs. Remote
There’s a constant tension between developing locally and running pipelines remotely. The trend is moving towards a hybrid model where developers leverage powerful local machines (e.g., Mac M2/M4) to run self-managed CI runners. This allows them to execute entire pipelines locally, speeding up feedback loops and reducing context switching. This shift brings back the importance of a powerful CLI and tools that can replicate the remote environment locally.
Optimization and Common Pitfalls
Several common mistakes hinder pipeline efficiency:
- Right-Sizing: Teams often neglect to optimize the resource allocation (CPU, memory) for their build jobs. Using insight dashboards and APIs to analyze usage can lead to significant cost savings and speed improvements.
- Underutilizing Caching: Making smarter use of caching and optimizing parallel builds are low-hanging fruits for faster pipelines.
- Miscalculating TCO: When considering self-managed runners, teams sometimes forget to account for the operational overhead (the Total Cost of Ownership), not just the infrastructure cost.
- Monolithic Pipelines: Like monolithic applications, large, complex, and hard-to-maintain pipelines should be broken down using a strangler pattern into smaller, independent, and more manageable units.
The Rise of Platform Engineering
As organizations scale, a ‘fracturing of DevOps’ is occurring, where most developers prefer not to manage operational complexity. This has led to the rise of platform teams that build and maintain the internal development platform. They provide standardized templates, enforce policies (e.g., with OPA), and manage CI/CD infrastructure as a product for other development teams, ensuring consistency, security, and compliance.
AI’s Impact on the Future of CI/CD
AI is poised to revolutionize CI/CD in several ways:
- AI as an Expert System: AI can act as an experienced advisor, analyzing pipelines to suggest optimizations, explain bottlenecks, and guide developers toward best practices.
- Natural Language and Automation: Developers will increasingly use natural language to generate pipelines. This will also enable self-healing and self-optimizing pipelines, but requires careful governance when interacting with production.
- CI/CD as a Guardrail for AI: A robust CI/CD pipeline becomes a critical safety mechanism. It allows development teams to experiment freely with AI-generated code, knowing the pipeline will enforce security, policy, and quality guardrails before deployment. This creates a vital feedback loop where an AI agent can learn from the pipeline’s checks and iterate on its code.