How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner
Harald Kirschner, principal product manager on VS Code at Microsoft, explains how a small team serving more than 50 million users moved from monthly to weekly releases by rebuilding its whole delivery system around AI agents. He covers faster shipping through agent-ready codebases, skills, and a 10x faster TypeScript Go build, quality practices like mandatory AI code review, AI triage, auto-fix PRs and staged rollouts, and faster learning through VSC-Bench evals and daily prototyping.
Rebuilding the delivery system around agents
VS Code shipped a monthly release for ten years, starting with version 1.0. Moving to a weekly cadence did not come from simply using more AI in daily work. It came from rebuilding the whole delivery system around agents. A small team serving more than 50 million users found that writing code was the easy part. The harder problems were review, quality, and figuring out what to build.
Measuring trust in agent-written code
The team tracks code survival, meaning the share of agent-written code that actually gets committed rather than thrown away. It started at 55% with GPT-4.1 and reached 86% with Claude Opus 4.6, driven by better harnesses and newer models. That number reflects growing developer trust, but success created new load. VS Code is one of the largest open-source projects on GitHub, and AI makes it easy to file issues, so volume rose with a mix of strong reports and automated low-quality noise. Open PRs grew from the internal team, from raw velocity, and from the community, and the number of community PRs merged also went up.
Shipping faster: an agent-ready codebase
The foundation is an AGENTS.md file: a lightweight map of the codebase that tells agents where to look. Teams generate a draft and then keep iterating on it, because these are living documents that should change as agents make mistakes. Teams that already invested in documentation and onboarding get the benefit twice, since the material that helps a new engineer helps an agent just as much.
Skills encode expertise. VS Code added an accessibility skill that packages the team’s best practices. Before, that knowledge lived with one person who had to be pulled into every review. Now the skill is maintained by the accessibility area owner and used by everyone. Expert knowledge that used to queue behind a few people becomes available to the whole team.
The litmus test is whether a PM can vibe code effectively in the VS Code repo. That means checking whether changes work out of the box, work locally, or break once shipped.
Speed also depends on the feedback loop. Moving to TypeScript Go, a native port, cut builds by 10x. With one developer, slow builds are annoying but tolerable because you can context switch while they run. With 10 or 20 agents hitting the same CI loop at once, every slow step compounds. A large early PR for an ask-questions tool became a polished, designed experience within weeks, because it gave the team a concrete idea to react to.
Letting agents use the application
Agents often report that a UI looks perfect when it doesn’t. The core principle is that if an agent cannot use your application, it cannot close its feedback loop. VS Code invested in two areas. The first is a component explorer, an automated build that screenshots every component on each change and highlights differences, catching ripple effects such as an icon disappearing or a layout shifting. Reviewers get visual diffs, and developers attach videos or screenshots to their PRs.
The second is a self-correcting loop. VS Code is essentially HTML running in Electron, so Playwright can drive it. A /launch skill opens VS Code in a given scenario, clicks through it, collects logs, and verifies fixes before and after a change, without a human clicking through anything. For iOS work, the speaker’s favorite was Xcode MCP, which lets an agent create and click through an iOS app from inside VS Code. Similar tooling exists for Android.
Quality: review, triage, and auto-fix
AI code review is now mandatory for every PR, after an initial period of skepticism. Review effort scales with the risk of the area of the repo (low, medium, or high), and humans do not look at a PR until the review comments are resolved.
Issue triage is handled by AI, which filters spam, enriches and translates reports, and assigns area owners. Humans still correct mistakes, for example through a Chrome extension for fixing duplicates, and those corrections flow back into the agent’s work. The point is to keep a human feedback loop in place as agents take on more of the work.
Error handling runs on VS Code’s own telemetry rather than an external tool. Roughly 51 billion raw events arrive per day. They are filtered down to exceptions with full stacks, grouped through fingerprinting, and sorted with ML, which reduces them to about ten issues filed and assigned to owners. An agent then investigates, finds the root cause (in one case, a cancellation request missing from the RPC protocol), and opens a PR. The pipeline mixes agents with deterministic systems, and humans still approve what lands.
Releases also changed. VS Code used to ship to 100% of users at once after testing. Because rollbacks are expensive for installed desktop apps, the team now stages rollouts and monitors error stacks and issues as they go. In the speaker’s words, they should have done this earlier.
Learning faster
VS Code runs its own evals, called VSC-Bench. New scenarios start as GitHub issues, and an agent bootstraps each one from a template. The team hill-climbs against these offline, then confirms gains with online experiments.
The evals also turned up a surprise. A trivial scenario asking a model to write a file containing hello world, five characters in total, cost the most expensive model 70x more tokens, and it was not the high-reasoning model. The lesson is that how a model behaves inside your harness matters more than its general reputation, so understanding your evals is critical.
Prototyping now happens in daily sprints. Instead of opening a PR to start a conversation, the team builds quick prototypes and iterates on them each day. Feedback that used to take a month now takes a day. Smaller squads with durable ownership keep these loops tight.
The broader takeaway is to tune the feedback loops, not just how much AI you use. Each success exposes the next bottleneck, and finding and fixing it is a continuous cycle.