Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

Denys Linkov details Wisedocs' journey of refactoring a complex, distributed ML pipeline into a monorepo, benchmarking the process against evolving AI coding tools. He honestly audits whether the six-month effort was justified, or if waiting for more advanced AI would have been better, ultimately concluding that the significant social and technical benefits made the refactor a successful strategic move despite current LLM limitations.

August 8, 2026 6 min read

Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs — source: https://www.youtube.com/watch?v=7vn4WpqNpck

Denys Linkov’s talk addresses Wisedocs’ journey of refactoring a critical Machine Learning pipeline, initially distributed across over ten repositories, into a monorepo. This complex system processed medical claims from PDFs exceeding 10,000 pages, some larger than video files. Facing issues like slow customer throughput, complicated updates, and a codebase nobody wanted to touch, a six-month refactor was initiated. The core question posed is whether this refactor was the right decision, especially considering the rapid advancements in AI coding tools.

The Challenge of Technical Debt and Modern Software Quality

Linkov likens technical debt to financial debt, emphasizing its compounding nature and the need for a clear Return on Investment (ROI). While AI engineering has shown immense progress in shipping velocity (citing Anthropic, Spotify, and Stripe), there’s a paradox: despite faster delivery, product quality, maintainability, and reliability have often degraded. He notes that uptimes for some leading companies fall below 3 or 4 nines, indicating a potential trade-off between speed and stability.

The Refactor Process and Evolving AI Assistance

The refactor officially began in April with earlier pre-work. Linkov details several phases and how AI assistance evolved during this period:

Orchestrator Evaluation: A two-month process involved evaluating five open-source orchestrators against 17 criteria, using a team of three to build Proofs of Concept (POCs). Initially, this relied heavily on manual work and basic AI. Linkov estimates that with current tools like “deep research” and agentic workflows, this process could be 90% faster, though he cautions against “AI psychosis” where reports don’t match product reality.
Benchmarking Refactor Tasks with AI:
- Initial Attempt (o3): A specific refactor task (implementing Temporal workflows) given to o3 took 3 hours of back-and-forth chatting within Cursor and resulted in 10 major mistakes. It was a manual, intervention-heavy process.
- Modern Models Rerun: Rerunning the same task with newer models showed significant improvement:
  - Sonnet 4.6 solved the task with just one additional iteration.
  - Opus 4.8 “essentially got it in one pass.”
- The interaction also changed, with modern harnesses providing more substantial tool calls, sub-agents, plan calls, shell commands, and verifications, making the process “a lot less manual.” Overall, this task would now take around 1/5th of the original time.

The “Agent Problem” and Task Accuracy

Linkov highlights a common frustration: kicking off an AI agent only to realize the prompt, plan, or requirements were incomplete. He references the METR task length curve, arguing that while it’s often presented at a 50% success rate, a more useful metric for practical application is the 80% or 90% accuracy rate. He argues that “if you’re kicking off a process that is going to take an hour and it has a 50% chance of completing, there’s a very high chance you just wasted that hour and you could have been doing something different.” He points to Anthropic’s Mythos preview, noting that even frontier models show significant decline in success rates at the 4-hour mark, and even 15-minute tasks can be inconsistent. This underscores the need for robust frameworks and primitives for “good agentic development.”

Outcomes of the Monorepo Refactor

The monorepo transition brought substantial benefits:

Increased Velocity: After rebuilding to parity with the previous ten-repo setup, the “steepness of that curve is immense and it didn’t slow down.” Commit velocity and code output significantly increased.
Developer Engagement: Previously, “nobody actually wants to touch the code.” Post-refactor, “almost every developer within the company is committing to this new monor repo,” even in areas outside their primary expertise, indicating improved developer satisfaction and reduced friction.
Business Impact: The refactor led to reduced pipeline execution time and costs, support for larger file sizes, and the ability to ship features that previously took months in “under a week.”

LLM Zero-Shot Refactor Experiment

Linkov conducted an experiment to see if a modern LLM could “zero-shot” the entire refactor. He tasked GPT 5.5 Extra High with refactoring the codebase. It “completed its goal in 10 minutes and 22 seconds” and wrote 2,000 lines of code. However, upon inspection, it “just implemented a bunch of scaffolding and didn’t implement the models,” even admitting in its output: “I did not add a raiseer deployment or bootstrap command yet.” This demonstrates that while models are improving rapidly, they are not yet capable of self-validating and “oneshotting” complex, large-scale refactors, though Linkov believes this capability might be “getting close” within six months for “pretty substantial refactors.”

Was the Refactor Worthwhile?

Considering the trade-offs—doing the refactor now versus waiting for more advanced AI tools—Linkov firmly concludes: yes, it was worthwhile.

Arguments for Doing It: While future AI will make refactoring easier, “AI native” code, if not properly managed, can quickly become new “legacy code” with low performance and quality, making future adjustments difficult. Guardrails are essential.
Justification: The refactor enabled Wisedocs to achieve business goals, accelerate development, improve pipeline efficiency (time, cost), handle larger data, and significantly boost feature delivery speed. Crucially, it transformed developer morale and engagement, making the codebase a desirable place to work.

Q&A Insights

Monorepo vs. Multi-repo for AI: Wisedocs moved to a monorepo. While models are improving at navigating multiple repos, end-to-end testing, verification, deployment, and sandbox setup remain significantly harder with distributed repositories.
PR Reviews: During the refactor, human PR reviews were crucial for context building and knowledge sharing, supplemented by local AI checks. The process is becoming more autonomous over time.
Future of AI-Assisted Refactoring: In six months, Linkov expects models to handle greater “complexity of the task,” moving beyond simple file rewrites to encompass faster research, POC work, validation, and hidden assumption checks for refactoring.

Linkov encourages a layered approach to AI delivery systems, stressing the importance of business value and trade-offs. He concludes that “models will continue to get better, but sometimes it’s good to pause, build a monor repo, and forge ahead.”