We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
Elie Bakouch, a research engineer at Prime Intellect, reports on setting Claude Code and Codex loose on the community Optimizer Speedrun, a race to train a GPT-2-level model in the fewest steps. Both agents beat the human record, but they behaved very differently, and none of them invented a new optimizer. The talk closes with a proposed AlphaEvolve-style discovery loop and an argument for doing this research openly.
Why test automated AI research in the open
The speaker frames the talk around a gap he sees in the discourse. Big labs keep saying recursive self-improvement, meaning models training models without human intervention, is coming soon, but there is no benchmark to check that claim, and especially no third-party benchmark from outside the big labs. He also argues that this matters beyond AI research itself. Much of the scientific work in the coming years will rely on AI tools, so understanding how frontier models actually do research is important more broadly.
From GPT-2 in 90 minutes to the Optimizer Speedrun
He traces the lineage of the benchmark. Andrej Karpathy made a video training GPT-2 from scratch in roughly 90 minutes, a feat that would have taken weeks not long before. The community then took that repo into modded-nanogpt, led by Keller Jordan, and pushed the time down to 45 minutes and then to under two minutes for reaching GPT-2-level validation loss. The speedrun is a game: reach the target loss in the least wall-clock time, with only the training and validation data fixed.
The Optimizer Speedrun is a variant released a few months earlier, where only optimizer-related choices can change, for example swapping Adam for Muon or another favored optimizer. The speaker notes this makes the task less about making code fast and more about finding the best method regardless of how much compute it consumes, which is closer to research.
Why speedruns make good environments
Speedruns are, in his view, both a good evaluation and a plausible training environment. The reward is positive when a run beats the current record and zero or negative otherwise. Runs are fast, with each taking about 15 to 20 minutes and records sitting near two minutes. The rules are clear and verifiable, which he thinks makes the setting suitable for genuine discoveries, since claimed improvements can be checked.
The experiment setup
His team pitted Codex (GPT-5.5) against Claude Code (Opus 4.8) on the community leaderboard. The agents ran freely on their Slurm cluster and were periodically stopped and restarted, which the team labeled V1, V2 and V3. By the third version, the agents were no longer holding the best record, so the team pointed them at the most recent human records and asked them to improve on those. This worked. A separate novelty track required records built only from new ideas, which proved considerably harder for the models.
The scaffolding was deliberately minimal. A goal.md file defined the rules and the agents read it, proposed ideas, and submitted jobs with sbatch on preemptible permissions, meaning other cluster users could cancel them. The agent then read the training logs and decided whether a run counted as a record. Validation required passing a statistical threshold to make sure an improvement was real optimization and not noise.
Two very different working styles
Claude Code repeatedly stopped every nine or ten hours, declaring that the record could not be beaten. The speaker then had to restart it and tell it to explore a new direction. Roughly a third of its time was idle, because there was no good way to monitor it mid-run. Codex almost never stopped, rarely asked questions, and was very impressive in its sustained autonomy.
The behavioral gap extended to the agents’ scratchpads, which served as their active memory. Codex wrote far more to its scratchpad, spawned more sub-agents, and burned far more tokens, on the order of a billion in total though much of that was cached input. Its roughly 250k-token context window forced frequent compaction, whereas Claude compacted about once per hour. The notes also differed in tone: Claude was excited and emoji-laden when it found a record, while Codex wrote in a terse, robotic, decision-by-decision style. The speaker normalized the plots by the number of active sub-agents to show that the difference was in behavior, not just in how much work was done.
Results
The human record at the time was about 2,990 steps. Claude beat it by roughly 50 to 60 steps and Codex by about 20. Codex was very fast in the early stage of the run. Both agents could fetch the latest human records at any time, and when restarted they pulled new records and improved on them, so the results reflect a loop that builds on community progress.
Toward a real benchmark
The speaker is clear that these results are informal and not yet publishable as a benchmark. A proper evaluation needs multiple seeds and identical conditions for every model. His team is designing three tracks: one with only the knowledge in model weights, one with access only to arXiv papers, and one with full access including the latest human records. These would run on both the original nanoGPT speedrun and the constrained Optimizer Speedrun.
Six days of agents and the token question
In a longer run of almost six days, Codex, Claude and Kimi were all effective, while GLM was still iterating when the talk was given. Claude improved in a steady, progressive way, while Kimi showed step-function behavior, with a breakthrough on day four that beat Codex’s record. Measured by elapsed time, the ranking looks one way, but measured by output tokens the story changes. Claude’s max mode consumed far more tokens than Codex and Kimi, and Kimi proved especially token-efficient.
How models use the literature
The models differ in how they engage with research papers. Codex did extensive paper search and found a paper no other model had surfaced, and that paper led to the best record of the run. The speaker finds this a telling example of literature search directly driving progress.
No novel optimizers
The speaker’s most sobering finding is that he expected the agents to produce some striking new optimizer, and they did not. Instead they combined existing papers and made incremental, plus-one improvements across many methods. No genuinely new optimizer or mechanism emerged. He treats this as a meaningful signal: the task is not simple, but it is accessible to human researchers who spend days or weeks on it, and the models could not find new mechanisms.
A proposed discovery loop
Inspired by AlphaEvolve and subsequent work, he describes a multi-agent loop. Generator models, both closed and cheap open-source ones, propose ideas. Each idea is run through the speedrun to obtain a reward. A judge then gives qualitative feedback and can apply a sense of taste about whether a method is promising. Promising methods are scaled up to more parameters and more tokens, because many speedrun methods are said to fail at scale, and scale has to be part of the loop. Humans remain useful for judging ideas and steering agents. He also suggests defining multiple speedruns with different objectives and constraints, which creates diversity and pushes search in specific directions.
What Prime Intellect is building
He outlines work in progress and recent releases. GPU sandboxing is in development so models can iterate in isolated environments, which is essential for this kind of work. The team is building its own efficient agents for an RL framework, with a file system, file read and write, and programmatic tool use, and is training a model for these tasks on top of open-source bases. A released library called Verifiers allows training and evaluating arbitrary environments on any RL setup, including large models such as GLM 5.2.
Why this should happen in the open
He closes by arguing that recursive self-improvement should be studied in the open. Many capable researchers work outside the big labs, and the goal is to make it easy for them to understand how models do research and to contribute to it.