The State of AI in Software Development: Data from 400+ Orgs — Justin Reock, DX
Justin Reock of DX presents five trends from research on about 200,000 engineers, covering deployment frequency, change failure rate volatility, the maintainability and change confidence gap, growing PR sizes, differences across junior and senior engineers, the DX AI Measurement Framework, platform AI readiness, agent experience, and why code generation was never the bottleneck. Case studies from Morgan Stanley, Zapier, and Spotify illustrate AI applied across the SDLC.
Justin Reock, Deputy CTO at DX, opens by framing the talk as a preview of DX’s quarterly State of AI report, built on data from roughly 200,000 engineers collected by a research platform created by the team behind the DORA metrics, SPACE framework, and DevEx framework. He stresses that the session looks at organizational trends rather than raw report figures, and he cautions that velocity metrics such as PR throughput and deployment frequency are proxies for how work flows, not measures of the value it generates.
Velocity: Deployment Frequency Rises, Perceived Speed Stays Flat
Deployment frequency, one of the four DORA metrics, is steadily increasing, although the curve is tapering after an early surge driven by engineers submitting PRs more often. Justin notes that this captures only one slice of the PDLC. It says nothing about revert rates, defect ratios, or change failure rate. Regionally, North America is trending upward while Europe pulled back in the last quarter. He attributes the differences to how work is organized, how AI spend and token costs are handled, and regulation.
Perceived delivery speed rose only about 4.5% over a full year of heavy investment, which Justin finds striking. He contrasts this with the METR study, a small study of 16 engineers whose follow-up acknowledged flawed signals. In that study measured productivity fell about 19% while perceived productivity rose about 20%, a spread of roughly 40 points. In DX’s aggregate data, perception stays largely flat, so the feeling of speed does not reliably track reality.
Quality: Volatile Change Failure Rates and a Confidence Gap
Change failure rate is where Justin says the impact is most volatile. Each line on the chart represents a company, and the goal is to sit at the bottom. Some organizations saw their rates rise by as much as 2 points against an industry benchmark of about 4%, which he translates to roughly 50% more defects shipped. He is careful to say the pattern predates AI and depends heavily on release pipelines and automated testing. What AI changed is the amplitude of volatility, which has reached extremes not typically seen.
Perceived maintainability rose almost 4%, while change confidence fell 6%. Justin calls this an interesting tension. Agents make code easier to understand and modify, yet engineers trust the outputs less and are more afraid of breaking things than they were a year ago. He describes this as a psychological effect that deserves attention.
PR Size and Incremental Delivery
Average PR size grew from about 44 lines to 72 over the study year, and Justin considers it one of the most important metrics to watch this year. He offers several explanations. Models are trained to produce average code. Build pipelines that take 45 minutes to an hour push engineers to batch AI-generated functions into one large PR instead of several small ones. Every extra line is a potential bug, a potential vulnerability, more to review, and less portable code. Perception of incremental delivery, one of the lowest-ranked developer experience drivers, dropped about 10%. That matters because small changes are what make rollback and review manageable.
Demographics: Juniors Adopt Most, Seniors Save Comparable Time
Junior engineers use AI the most, which Justin attributes partly to having less to unlearn when entering the industry. Agent experience data, collected by asking agents about the tasks they worked on and the tokens they consumed, shows juniors spending more tokens on the same use cases, which he describes as a normal learning curve. Staff-plus engineers, however, save roughly the same amount of time while burning fewer tokens. They are better at spotting hallucinations and understanding the architecture around their changes. Aggregate time savings are therefore flat between these groups. Smaller companies lead in time savings because simpler release pipelines and less organizational friction give them more agility.
Measurement: The DX AI Measurement Framework
Justin argues that measuring developer productivity and experience was already difficult before AI, and that AI complicates it further by confounding several variables. Foundational metrics remain the trusted anchor. The real question is how AI affects quality, speed, and value generation. The framework has three dimensions. Utilization covers daily and weekly active users and similar signals. Impact asks which business metrics should move as utilization rises. Cost is pressing, because token spend is growing quickly. He describes a maturity curve in which most organizations start with utilization, then correlate user cohorts against impact metrics such as PR cycle time, PR size, and review pushback. A DX white paper details the methodology.
Platform AI Readiness and Agent Experience
Justin summarizes the arc as follows. In 2024 everyone got a coding assistant, in 2025 teams started building agents, and now organizations are discovering their infrastructure was not ready. The readiness indicators he lists are clear and accurate documentation, data structures with straightforward relations, manageable modular code, reliable local CI, and non-flaky test suites. He points out that these are simply the classic markers of good developer experience. What helps humans also helps agents, so organizations may finally make the investments they should have made over the last couple of decades.
Agent experience uses qualitative feedback from the agents themselves about where they hit friction with humans, lacked context, or had to cycle through repeated feedback loops. Breaking this data down by use case shows which tasks yield the most efficient token spend and the most value creation.
Fifth Trend: Code Generation Was Never the Bottleneck
Even if engineers received perfect, instant code, Justin notes that this would address only about 14 to 16% of the overall value stream. The median gain in PR throughput in the study was about 7.7%, with a 13% average. Even top performers reached roughly 70%, and nobody hit 2x, 5x, or 10x. Time saved on AI is being outweighed by meetings, context switching, dev environment friction, and other non-AI factors. Citing Eliyahu Goldratt’s theory of constraints, which inspired The Phoenix Project, he argues that an hour saved away from the bottleneck is worth nothing. Organizations must find the real constraint and fix it.
Justin then walks through case studies that show AI applied across the SDLC.
- Morgan Stanley’s DevGen.AI agent interprets legacy code such as mainframe Natural and COBOL and generates PRDs so engineers can skip the reverse engineering step. It saves the firm about 300,000 hours a year.
- Zapier built an agent ecosystem that handles administrative overhead such as standup summaries, cutting standups from five times a week to two. It onboards engineers in about two weeks, against an industry norm of more than a month. Most notably, it sees about 15% more value per engineer and is hiring more, which Justin frames as a throughput and innovation-capacity story rather than headcount replacement. Its agent also automates roughly 3,000 code reviews a week, leaving superficial findings in PR comments as a record that human reviewers can build on.
- Spotify’s SRE incident agent gathers remediation steps from runbooks and incident context, then posts them to communication channels so responders start with context instead of spending minutes on discovery.
Justin closes by pointing to the Q1 report, which already contains many of the session’s findings, and the Q2 report, expected around the end of July, with a preview in his afternoon talk.