From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer
Rafal Wilinski of Runlayer argues that when one agent solves a hard problem, that knowledge should become a skill shared across the whole company. He covers why skills matter more as agents run longer, three problems with current skills, using MCP as a governed distribution layer, distilling skills from successful and failed agent traces inspired by the Voyager paper, and the organizational flywheel as a durable competitive advantage.
When one agent finally cracks a hard problem, why shouldn’t every agent in the company learn it too? Rafal Wilinski, founding engineer at Runlayer, argues for moving from a single self-improving agent to a self-improving company, where agent learning is combined with network effects.
The playbook that rarely gets written
He opens with a familiar image: the best engineer, salesperson, or lawyer you ever worked with, who every time they cracked a genuinely hard problem wrote down exactly how and handed the playbook to the team. In practice this almost never happens. Once a session closes or the engineer moves on, the debugging and reasoning behind a breakthrough disappear. His claim is that the primitives to fix this already exist, and the talk is about assembling them.
What a skill is, and why it matters more now
He scopes the talk to skills among the levers for improving agents, alongside better models, prompting, and harnesses. A skill is a playbook the agent can read when it judges the knowledge relevant. The mechanism is progressive disclosure: the client first sees only a short name and description, and when the agent decides a skill is useful it loads the full contents into context and changes its trajectory accordingly.
The reason this matters is that models have improved quickly. He points to the METR time-horizon benchmark, which measures how long agents can work independently while keeping a reasonable chance of completing the task. The trend is clearly upward, and agents now handle deeper multi-step work. But longer horizons cut both ways. A capable agent can dig a very deep rabbit hole, spending hundreds of tool calls defending a wrong thesis for hours, and a subsequent agent may repeat or worsen the mistake. The early trajectory matters a great deal, so skills act as guardrails that point agents at the right target from the start, saving tokens and money.
He describes the most valuable skills as the rituals nobody wants to rediscover. Examples include the exact sequence of scripts and feature flags for a clean rollback, or the precise language to use with an enterprise customer who is about to churn. This knowledge is rarely glamorous and is usually embedded deeply in the company. If agents must reconstruct it from scratch, they will often fail, and the failure can be expensive. He suggests this is a major reason many AI proofs of concept fail: the agents simply lack the company’s playbooks.
Three problems holding skills back
- They are local-first and developer-centric. Skills tend to live as markdown files, JSON, CLI commands, and git repositories. That works for developers but leaves out marketing, HR, legal, and finance teams, which is a problem for an AI adoption lead trying to roll out a strategy company-wide. He shows a meme of a non-technical person running the npx skills install command, and warns of the risk: a single install can bring prompt injections, a script that requests root permissions, and someone else’s API key. Runlayer scans thousands of skills daily and finds ‘really weird stuff.’
- Nobody has time to write them. Reflecting on a hard problem and documenting it for posterity takes time, effort, and intention, so most problems never get a playbook.
- Client support is uneven. Cloud ecosystems tend to support skills well, but the chat clients used by non-engineering departments often do not.
MCP as the distribution layer
When skills launched, some people declared MCP dead, arguing that they could just hardcode CLI commands in markdown. He thinks this missed the point. Tools provide capabilities, while MCP can also deliver knowledge. The proposal is a remote MCP server acting as the single source of truth for the company. Clients connect and search for skills, and the server enforces policy and serves the best version of a skill based on the query and the user’s assigned role. Runlayer has run this internally for about six months.
There is already a Skills Over MCP working group. The official spec leans toward MCP resources, but support for resources is inconsistent across clients, so Runlayer delivers skills through tools. This addresses the first and third problems: MCP works in nearly every client, not just developer tools, and it gives a VP of AI or adoption lead one governed place to manage what knowledge gets distributed.
Self-improving agents and the Voyager lesson
He credits the Voyager paper from 2023, written shortly after GPT-4 was released. Voyager dropped an agent into Minecraft with no instructions and let it set its own goals. The agent improved without any changes to weights or prompts. The key ingredient was a skill library. When the agent discovered something, it saved the procedure as code, specifically the sequence of tool calls that led to the discovery. Later, when it needed a crafting table or a diamond pickaxe, it retrieved the recipe rather than solving the problem again. The core idea he takes from this is that exploration should be turned into reusable procedural knowledge.
Runlayer applies a similar approach. Agents do real work, and when a run is interesting, whether it succeeds or fails, the trace is passed to a frontier LLM that distills it into a skill. He argues that failures teach the most. A run that goes sideways reveals missing guardrails, missing libraries, and edge cases nobody anticipated. Successes make a skill more trustworthy, while failures make it more robust. This happens asynchronously and without a human in the loop.
Results: from 36% to 100%
In the test, the team asked Claude Sonnet 4.6, a capable but not top-tier model, to run a Chromium fork inside AWS Lambda. It succeeded only 36% of the time, which is not production-ready. They turned on self-improvement. Once one invocation cracked the problem, the distiller produced a reusable skill that was included in all future runs, and the success rate rose to 100%.
He emphasizes that success rate is the easiest number to show, but a good skill does more. It narrows the search space, so the agent makes fewer tool calls, explores fewer branches, and spends fewer tokens and less money.
Rented intelligence and the organizational flywheel
He argues that frontier intelligence is rented, not owned. Models can be deprecated or can seem to get worse, and people routinely complain that a model is ‘dumb today.’ While you have access to the best version, you should use it to discover and persist procedures, building a catalog of how work actually gets done in the company, something like a collective subconscious.
The pieces are already in place: skills to retain procedures, MCP to distribute them, and a central knowledge base that is continually updated. Scaled across every agent and client, the system works as follows. Similar runs are grouped and distilled into a skill. The skill passes checks for things like prompt injection and PII, lands in the skill library, and is served through the MCP gateway to every client and agent in the company. The result is a constantly evolving knowledge base. Once one agent solves a hard problem, that solution persists and is available to everyone, so no one has to rediscover it.
Your real moat
He acknowledges a common anxiety: models are becoming commodities, the cost of software is falling, and everyone at the conference seems to be building the same product. His answer is that the moat is the flywheel itself, a distilled, governed, and compounding record of how your company operates. It raises the floor for everyone, makes the company resilient to model deprecation, and lowers costs without changing models. A competitor cannot copy your internal culture and accumulated identity.
He closes by inviting discussion on skills, MCP, and agents, and says that Runlayer is building the AI golden path for enterprises.