Devin 2.0 and the Future of SWE - Scott Wu, Cognition
Scott Wu, CEO of Cognition AI, discusses the exponential growth of AI capabilities in software engineering, likening it to a "Moore's Law for AI agents" with a doubling time of every 70 days. He chronicles the evolution of their AI agent, Devin, from handling repetitive code migrations to autonomously managing entire backlogs, highlighting the key technical challenges and paradigm shifts at each stage.
The capability of AI agents in software engineering is doubling every 70 days, a rate described as a Moore’s Law for AI agents. This exponential growth, equivalent to a 16-64x improvement annually, has rapidly advanced the scope of tasks AI can handle. Eighteen months ago, the state-of-the-art was simple tab completion. Today, AI agents like Devin can function as autonomous engineers, performing tasks that take humans hours. With each doubling of capability, the primary bottleneck and the ideal human-AI interface change, leading to a rapid evolution of tools and workflows.
Summer 2023: Repetitive Migrations
The first consistent, high-value use case for AI agents was handling large-scale, repetitive code migrations.
- Tasks: Upgrading frameworks (e.g., JavaScript to TypeScript, major Angular or Java version bumps) across thousands of files in a codebase.
- Challenge: These tasks follow a clear, step-by-step process but are too nuanced for a simple deterministic script. The primary challenge was reliable instruction following.
- Solution: A “playbooks” system was developed to outline a precise set of steps for the agent to execute. Additionally, a knowledge and memory system was built to incorporate human feedback, allowing the agent to learn from past interactions and improve its performance on subsequent tasks.
Fall 2023: From Isolated Bugs to Multi-File Changes
The next step was moving from rote migrations to more general software engineering tasks, starting with small, contained changes and evolving to more complex ones.
- Intern-Level Tasks: Initially, Devin could handle isolated bugs or feature requests affecting one or two files, such as modifying a UI dropdown. The main technical hurdle was creating a stable and reproducible environment. This was solved by building a system for clean, remote VMs with snapshotting and rollback capabilities, able to run linters and CI checks.
- Broader Bugs & Features: As capabilities grew, Devin began tackling issues that required working across multiple files and understanding broader codebase context.
- Challenge: Making consistent changes across hundreds of lines and several files required a deeper understanding of the code’s structure.
- Solution: The agent was enhanced to use more than just the text of the code. It began leveraging language servers, call hierarchies, and Git commit history to build a richer representation of the codebase. This enabled direct integration with tools like Slack, where a user could tag Devin on an issue and have it build the solution.
Spring 2024: Collaborative, Complex Task Solving
For more complex or ambiguous tasks, the paradigm shifted from a simple prompt-and-execute model to a more collaborative and iterative workflow.
- Tasks: Open-ended problems like architectural improvements, performance profiling (“this function is slow, figure it out”), or designing complex error-handling logic where the solution is not known upfront.
- Challenge: A simple text prompt is insufficient for these tasks. The agent and the human need to collaborate to explore the codebase, define the problem, and scope the solution.
- Solution: This led to the development of the in-IDE experience (Devin 2.0). New tools were introduced to facilitate this collaboration:
- Deep Wiki: A feature that exposes Devin’s internal representation of the codebase, allowing humans to understand its “thinking” and ask questions.
- Codebase Search: The ability to ask natural language questions about the code to quickly understand different components. The workflow became more interactive, with the human closely monitoring the initial 10-20% of the task (scoping and planning) before letting the agent work autonomously on the execution.
June 2024 and Beyond: Autonomous Backlog Management
The most recent evolution is the ability for Devin to autonomously work through a backlog of tasks from project management tools like Jira or Linear.
- Challenge: To “kill the backlog,” an agent needs to not only execute tasks but also scope them, decide when to ask for human clarification, and, most importantly, test its own work.
- Solution: This required building two key capabilities:
- High Confidence: The agent must be able to assess its own understanding of a task and decide whether it can proceed confidently or needs to request more information.
- Asynchronous Testing: For an agent to deliver a complete pull request, it must be able to write and run its own tests, interpret the results, and iterate on the code until it passes. This self-driven testing loop is crucial for autonomy.
The journey from single-line completion to project-level execution demonstrates that building an AI engineer is not just a text prediction problem. It’s a systems problem that involves human-computer interaction, planning, debugging, and integrating with a complex toolchain. The rapid pace of development is expected to continue, promising another 16-64x increase in capabilities over the next year.