The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI
Agents have multiplied code output, but human review cannot keep pace, and reviewer effectiveness collapses past roughly 400 lines. This talk by Laurie Voss surveys what OpenAI, METR, Cognition, Cursor, GitHub, Carlini, and Bun are doing about it, from mergeability benchmarks and multipass automated review to prompt injection and production monitoring as the last line of defense.
The review bottleneck
Developers using autonomous agents wrote 741% more code but shipped only 30% more software. That figure comes from a study of more than 100,000 GitHub developers, matched against telemetry showing when each one started using AI. Generation is no longer the constraint. Human review is, and its speed has not changed. Reviewer effectiveness falls off sharply past about 400 lines in one sitting. A 10,000-line agent pull request would take three or four working days to get a real human review, and a developer can now run a dozen agents at once.
“Just review harder” fails on both counts. It burns people out, and the numbers don’t support it. The best study on this, run at Cisco over ten months on roughly 2,500 reviews covering about 3.2 million lines, found that reviewers stop finding defects effectively past those thresholds.
Stop reading the code?
Peter Steinberger, creator of OpenClaw, argues that developers should stop prompting coding agents directly and instead design the loops that prompt them. Andrej Karpathy has made a similar argument about taking yourself out of the loop, because the human in the loop holds the system back.
OpenAI published the most concrete account. In February, it described an internal product built with no manually written code. It started from an empty repository, and agents wrote everything, including the scaffolding that reviews the agents. Five months later the codebase had about a million lines and roughly 1,500 merged pull requests, built by three engineers. Humans may review pull requests but are not required to, and nearly all review effort has moved to agent-to-agent. OpenAI did not release the product or explain how it works, which suggests there are still holes in the approach. The underlying bet is that loop design can substitute for inspection, and whether that holds depends entirely on what the loop can see.
Passing tests is not mergeability
For years, the industry’s proxy for quality has been whether tests pass, because that is what benchmarks measure. METR tested that proxy directly in March. It took pull requests that had passed SWE-bench and had four active maintainers of those same open-source projects review them. Only about half were good enough to merge. The failures were not about correctness. They were about code quality and changes that quietly broke code outside the test suite. METR noted that the agents got no chance to iterate on feedback, but that caveat amounts to putting a human back in the loop.
Cognition’s FrontierCode takes the maintainer’s question, “would you merge this?”, and turns it into a benchmark. More than 20 maintainers built 150 tasks from their own repositories, each representing over 40 hours of expert work. Grading covers behavioral correctness, regression safety, scope discipline, test quality, and maintainability. It is, in effect, a human review rubric made machine-checkable. One leading model scores 88% on SWE-bench Pro but only 29% on the hardest slice of FrontierCode, a 51-point drop on the same job when the question shifts from “does it pass?” to “would you merge it?” GPT 5.5 scores under 6% on the same tests. The strongest models are nowhere near passing human review reliably.
Mergeability as a training signal
Sarah Guo has argued that compilers and test suites are free verifiers, and anything cheaply verifiable can be trained against until models beat it. That is why code generation improved so fast. A benchmark of genuine mergeability would therefore be more than a measurement. It would immediately become a training signal for frontier models, covering maintainability, scope discipline, and regression safety once reliable tests exist for them. As the speaker puts it, “whoever writes today’s review standard is writing next year’s default model behavior.”
There is precedent. In 2024, OpenAI trained CriticGPT to catch bugs in model-written code, and human reviewers aided by it beat either the model or the humans alone. That signal was folded into later models almost instantly. Mergeability standards, by contrast, are still unwritten. No one has a good rubric that captures everything a human weighs when deciding whether a change is mergeable.
Automated review in production
GitHub’s Copilot reviewer has done 60 million reviews and accounts for more than one in five code reviews on GitHub, making machine review the mainstream default on the largest code host.
Cursor has published its reviewer architecture, and the details show what the job really is. The first version ran eight review passes over each diff, shuffling the reviewer order between passes because order changed outcomes. The purpose was filtering false positives, since a reviewer that flags good code as bad will get ignored. A team at Peking University tested the same multipass idea independently and found it raised review quality by up to 44%. The rebuilt Cursor reviewer reasons over the diff, calls tools, and decides where to dig. The most telling change was that the team had to tell the model to be more suspicious. Left alone, it looked at the code, said it looked good, and shipped it, which is what a human might do too. Assuming something is wrong by default is, in the speaker’s view, most of what a good review is.
Review is also fusing with repair. Cursor’s reviewer now spawns a fix agent from its own findings, handing back a patch to approve, and it runs code to prove its bug reports are real. The line between reviewing and rewriting is getting thin.
Other vendors are crowded into the space. CodeRabbit has reviewed over 13 million pull requests. Greptile builds a graph of the whole repository so reviews can trace how a change lands in distant code. Graphite builds its evaluation set from which of its suggestions developers accept or reject. Every one of them measures success the same way: whether a human accepts the answer. Cursor calls this its resolution rate and has pushed it from 52% to over 70%. These systems are being tuned daily against human acceptance judgments, which previews what models will do.
Can you skip the human?
Anthropic’s Nicholas Carlini had 16 agents build a C compiler in Rust across about 2,000 sessions with no human in the loop, and it compiled the Linux kernel. Carlini’s own warning was that it is easy to watch the tests pass and assume the job is done, and that it rarely is. The test harness and feedback systems that governed the agents were written by humans, so there was no human in the loop but there was a human on the loop.
Bun’s port of its runtime from Zig to Rust was about a million lines written by agents in six days, and no human read the diff. The existing test suite passed at 99.8%, so the tests did real work. But the ported code has 13,044 unsafe blocks, where the author asserts rather than proves that memory is handled correctly. A comparable human-written Rust codebase of that size would have about 74, roughly three orders of magnitude fewer. Tests certify behavior at the public interface but cannot check thousands of assertions they were never designed to look for. The humans are out of the loop, and what lies under the surface is unknown.
OpenAI’s experience shows what skipping review costs. Codex reviews its own changes and calls in more agents to review those reviews until every reviewer is satisfied. It is made bootable for each change so it can run a copy of itself and inspect the UI, and the logging stack is exposed to the agent. Even so, for a while the team spent every Friday cleaning up AI slop by hand, until they trained agents to find and remove it. Review did not disappear. It was rebuilt as a system, and humans built that system.
Dex Horthy spent six months telling people not to review code and ship whatever the agent produced. In March he reversed course on stage: “Please read the code. We tried not reading the code for like six months. It did not end well. We had to rip out and replace large parts of that system.”
Where the human checkpoint survives
Tests cannot see context outside the suite. They don’t reveal that a module has three external users, or that a cron job nobody will admit to writing depends on it. The human checkpoint survives in predictable places: where correctness is not cheaply checkable, where the blast radius is large, and wherever someone has to put their name on the result. In security-conscious environments, proposals to remove human review get an immediate no. The human role is moving up the stack, from inspecting code directly to designing the systems that inspect it and defining what good means.
Who reviews the reviewers?
The answer is still humans. Anthropic’s automated security reviewer carries a caveat in its readme: it is not hardened against prompt injection and should only review trusted pull requests. A March study found that vulnerable code dressed up in an innocent commit message fooled an autonomous review agent in 88% of attempts, while the same attempts fooled human reviewers only 35% of the time. Taking the human out of the loop means losing the reviewer that was hard to fool. Automated reviewers accept confidently framed bad code, and agents are very good at producing exactly that. The field also disagrees on how to grade the graders, and benchmark scaffolds have been caught leaking answers.
Production as the last reviewer
With pre-merge review handled by machines, what the code actually does in the world becomes the last reviewer standing. Once code ships, test results stop being the interesting signal. The trajectory, meaning what the system did step by step when it ran against the real world, is what matters. Shipping automatically reviewed code makes production observability indispensable.
Code review is being rebuilt, not killed
Code review is changing enormously. It is being rebuilt as an engineered system, and humans are moving from the engine that reads code line by line to the pilots who run the system. Every layer of that system, including the benchmark, classifier, rubric, test suite, and eval, is itself unreviewed until someone takes responsibility for it. The teams that win the next few years won’t be the ones that generate the most code. They’ll be the ones who can show, with evidence, why they trust what they shipped.
What to do today
Stop reviewing pull requests line by line, which is the wrong level of abstraction. Human judgment is extremely valuable, but it can scale much further than it does now. Put effort into a reliable review harness: codify your definitions of good, your company context, and your domain knowledge, then have agents work within that. Gains come from concentrating effort higher up the stack, in reviewers, rules, and evals, rather than in reading diffs.