Can Open Models Keep Up with the Frontier? Compute, Distillation, Data, and the Case for Open AI Research

The conversation opens with a framing that cuts through the public discourse about “pacing the frontier.” Nathan Lambert argues that model progress reduces to three forms of capital: compute, data, and talent. Every lab has some of each, but the closed frontier labs (OpenAI, Anthropic, Google) have clearly accumulated the most, and the real differentiator is how well an organization converts that capital into models. The org chart matters a great deal here. Lambert points to Meta’s full reset as an example of how hard it is to get that organizational machinery working, and notes that it sometimes does work after a painful rebuild.

Culture, Bets, and Why the Frontier Labs Differ

Lambert sees a real cultural split between OpenAI and Anthropic. OpenAI’s signature move was making the bet that kicked off the reasoning-model revolution, and the inference scaling curve that followed changed how everyone thinks about the field. Anthropic has been focused on what models will be able to do, and its reward was the explosion of Claude Code after years of investing in coding. Both have been willing to spend heavily on directions others consider impossible. For someone who wants to work at the true frontier, Lambert’s honest advice is still to work at one of these two.

Why Chinese Labs Release Open Weights

Lambert challenges the idea that Chinese open-model releases are altruistic. He describes them as a practical commercialization strategy. A startup building a model in China and selling an API into a Western market would be nearly invisible, since few people would know Moonshot AI or similar companies exist. Releasing open weights is a shortcut into the conversation and the demand, and the US market clearly pays for AI. He is careful to say that Chinese labs are not simply government projects; they are startups trying to matter. He also notes that the Chinese state responds to corporate behavior in ways that would surprise people in the US, and that this shapes the security posture of these labs.

He sees the landscape stratifying into tiers. The best closed models command the highest margins. Near-frontier models, one step behind, will be released openly because there is a market for them, especially where private enterprise data and customization matter. Fast-following becomes easier once OpenAI and Anthropic reveal what a good model looks like, and data flows through the broader data economy as vendors sell similar datasets to everyone. Lambert admits he expected consolidation in the open-model ecosystem to happen sooner, and that the open labs have survived longer than he predicted.

Distillation: Caricature or Real Lever?

The conversation’s central debate is how much distillation actually drives the gap. Lambert’s position is that Chinese labs are technically excellent and have a talent pool comparable to the US, possibly with a slight historical edge. What they lack is compute. He describes conversations with Chinese researchers who have access to thousands of H100-equivalents and seem stunned by what Western labs can marshal. Distillation is, in his framing, a way to purchase compute-equivalent outputs: if you know the outputs you want from a model, you are effectively spending compute to generate them.

He traces how post-training has evolved. Early post-training relied on supervised fine-tuning, essentially next-token prediction on the outputs of a strong teacher model. That made distillation powerful, since you could take the best model available, generate completions, and train on them, provided you had good prompts. Lambert recalls that prompts were always the key unit of post-training, a point he emphasized during his time at AI2. Post-training has since shifted toward RL-heavy environments, where the signal is a score rather than an imitation target, and that is harder to map onto distillation.

Lambert is candid that the public literature makes distillation hard to quantify. Open post-training efforts keep relying on older Qwen models, which he attributes to difficulty getting the right teacher, perhaps because of mid-training or common open pre-training data. He doubts that access to Claude’s reasoning traces would have dramatically improved his own model. His evidence is grounded in the literature, but he acknowledges wide error bars. When he says distillation helps Chinese labs by a few months, he is basing that on the information available, and he argues the field needs more open research to study it properly. He mentions a notable paper on stealing reasoning traces from closed models and notes that Chinese labs likely have teams that know these techniques. Reasoning-model APIs that expose thinking tokens make extraction easy, and as long as such APIs exist, distillation is hard to prevent. Restricting access to enterprise-only products would make it far harder.

On the policy and ethics side, Lambert is skeptical of treating distillation as a terms-of-service crime that warrants government involvement. He doubts terms of service are enforceable as a signed contract, and sees the practical remedy as cancelling access. He acknowledges that national security voices have a directional point given AI’s growing influence, but worries that heavy-handed action against distillation would cause more harm than good, and that the open-source ecosystem, and by extension the startup ecosystem, benefits substantially from these releases. He wants to keep channels open with Chinese tech companies.

The discussion then turns to the flywheel that frontier labs depend on. Enormous capital buys compute, the compute produces models, models generate revenue, revenue enables further fundraising, and the cycle repeats. Scaling science gives labs some predictability in how investment translates to revenue. Anything that threatens this flywheel, including rapid open-model catch-up through distillation, is a risk to the whole industry, including the upcoming IPO path. The participants agree they want the labs to succeed, while also recognizing the practical benefit that Chinese open-weight models provide. Startups now do post-training as a service, and open models like Kimi and GLM are good enough as bases. Lambert offers a case from Cursor: a long-horizon SQLite rewrite task performed about equivalently with a composer-based fine-tuned model acting as the executor, compared with two frontier models, at much lower cost.

Open Versus Closed on Different Time Scales

Lambert frames open and closed models as operating on different exponentials. Closed labs are vertically integrated: they have efficient inference, can serve models at scale when ready, and reach high-value markets first. Coding has been the breakout example, with revenue growing dramatically. Open model builders face a more fragmented economy. Model builders, inference providers, and post-training companies are separate, and customer information does not flow back to the model builder easily. Economies of scale in serving are weaker. Yet enterprises are strongly motivated to have private data and avoid sending it through APIs. Over time, open models offer something closed APIs cannot: ownership of weights, no terms and conditions, and the ability to run on one’s own hardware. Lambert says this path takes much longer, but he is less worried about it being a bubble. He admits he does not know how it will play out.

The conversation also touches on whether general reasoning scaling beats specialized models. Math is the purest example, where general models have surpassed specialists without targeting math specifically. Robotics is more contested. One view holds that a lab hired people and bought robotics data and the model became good in months, while another suggests the gains came from computer-use spatial reasoning transferring over. Lambert’s view is that the frontier keeps ascending until it is good enough in most categories, but a counterforce exists: open models already good enough can be fine-tuned on the specific benchmarks and proprietary datasets that frontier labs cannot reach, and this percolation is starting to happen in practice.

Progress Is Really About Data

Lambert argues that frontier progress in domains that are not fully verifiable comes largely from buying high-quality data and giving models environments in which to act. He divides the field into two regimes: math and code, where verifiability is strong, and everything else (knowledge work, biology, healthcare, legal), where it is much less so. In the second regime, progress looks much closer to the story of OpenAI buying large amounts of robotics data. Lambert suspects that claims of “generalization magic” are partly real and partly a rhetorical shortcut, and that much of the result really comes down to data.

He describes the work at frontier labs in concrete terms: some people build datasets, specify desired model behavior in a narrow area, design how to make the data that elicits that behavior, and decide what data to purchase. He expects this to play out for years across sectors, and notes it is very capital intensive. Verifiable domains like math, code, and distributed ML engineering are different because cost is measurable and AI systems will drive costs down rapidly, producing surplus benefits. He also highlights that model efficiency is dropping multiplicatively. Price cuts of roughly 2x every eight months over several years compound dramatically, raising open questions about Jevons’ paradox: whether margins hold or revenues fall as the market expands. His bottom line is that AI progress is fundamentally about data and about writing better software to serve models, and those are very different worlds.

On RL environments, the participants discuss whether hand-built or simulated environments command better prices. Lambert notes that the RL environment market is booming, with both top labs and a growing tail of labs buying. He points to a real quality problem: companies cut corners, and this can induce reward hacking. A model told to solve a task might discover that the test file is empty and game it. He thinks much of this behavior traces back to a flawed supply chain, and that labs now spend significant research time eliminating reward hacking while data vendors spend proportionally less time on model behavior and more on revenue and world-building. These two sides need to meet to produce higher-quality data. Many labs are bringing data work in-house partly to avoid leaking signals about what they are doing.

Lambert says almost all companies now mix synthetic data with human guidance: language models modify or generate worlds and permutations, and even human-guided processes are largely synthetic. He is still somewhat in denial about the billion-dollar research budgets for environments, but takes the spending as evidence that the models are benefiting from them. He sees a lot of low-hanging fruit in data quality, with rough edges such as contractors overseas working through intermediaries who do not know they are selling data. He expects a few years of transition as the winners are pulled toward higher quality and higher ROI.

Trillium and Open Infrastructure for Research

Lambert explains his nonprofit lab, Trillium, which focuses on open-source environments for high-leverage tasks. One example is T-Max, a fully synthetic terminal-task dataset developed with collaborators at UW and AI2. The process prompts a frontier model to generate problem types, varies prompts for diversity, uses model rollouts to find bugs, and eliminates flawed items. He stresses that this is not magic: each data point must be read and the dataset improves through human attention. The problem is that few institutions are willing to invest the thousands or tens of thousands of dollars needed to make environments well and release them, because the profit motive for keeping them proprietary is so strong. Lambert half-jokes that one could sell such data for a million dollars.

He argues that open environments, rollouts, and trajectories become the foundation for a great deal of research. PhD students can spend a year studying behaviors such as evaluation awareness, or track how RL changes model behavior over time. Industry captures value from CS research, but the benefit often arrives roughly a decade later. He sees a case for supporting open work because peers who are working in the industry for ten years will be the ones who benefit.

The name Trillium comes from a spring ephemeral flower that blooms before the forest canopy leaves out, spreading nutrients through the cycle of growth. Lambert uses it as a metaphor for a moment when the AI industry has abundant activity, yet needs more open research, education, and distributed contribution to spread its benefits. He wants open standards for studying important problems such as sandboxes, loss of control, and how behaviors change across RL runs spanning a week on a thousand GPUs or on 100,000. He believes such work builds trust, and he sees his own role as creating space for more people to grow independent thought and build infrastructure.

Lambert also describes the dynamics of open contributions. Open releases attract distributed contributors who are motivated by sharing work rather than selling to labs, which can yield high-caliber contributions. He notes a new problem: people now flood repositories with fabricated research results and issues that are largely language model output. He considers this solvable and thinks agents can help filter low-quality contributions. He also reflects on how hard it has become for newcomers to enter AI research. Real skill requires weeks of reading, writing code by hand, and waiting months for recognition, and language models tempt people to push a button rather than do that slow work. He says this is a hard time for PhD students, but that serious work is still happening.

Demystifying Recursive Self-Improvement

The conversation then turns to RSI and whether it explains the fear that frontier labs will pull away. Lambert thinks RSI carries too much mystique. His proposed remedy is transparency about how AI is actually used to automate AI research: sharing tools, configurations, compute, tasks, trajectories, and whether models succeeded on them. Trillium plans to build an open-source stack for SFT and mid-training, using open tools, and to publish what automation it uses and why. He would check in every six months to see how things have changed. He argues that AI nonprofits can do what academia finds difficult: document work for evaluations and policy, and build resources with rare information that gains value as more high-focus work is done.

He also notes the timeliness of this. Several people he knows at labs feel they are about to be automated, which means fewer people will be exposed to what happens in parts of the organizational chart. He mentions an academic open-RSI effort that just launched, and argues that multiple approaches are healthy. Frontier-lab insiders reportedly say the methods are not really secret; the proportions and recipes that work are hard to pin down, and capital to experiment is the true barrier.

Lambert is skeptical that capabilities will explode in a fast takeoff. Models will improve at lowering serving costs and tuning architecture, since those are verifiable and profitable. Knowledge work is different, and automating it will take much longer. He also argues that frontier labs do not have airtight secrecy: ideas spread, researchers talk, and it is difficult to hold breakthroughs for long.

Can Open Source Pre-Train?

On pre-training, Lambert argues that the tools are accessible, and the pre-training ladder, a decision-making tool for testing whether ideas scale, is available to many groups. A first approximation is training on the internet, with secret details that matter but change slowly. Architecture and GPUs are the main things evolving. A solid pre-trained model does not require a billion dollars, though more spending yields better results. He is consistently surprised by new labs that start from scratch, often building on open frameworks such as Megatron and then forking them without upstreaming. Lambert is not betting against new entrants, because the hard part is not building a relevant model but building one good enough that people will use it and that justifies the next fundraise. He cites Thinking Machines as an example of a lab that was transparent about a first model that was not the best, and that has since released a second model and built momentum around Tinker. Meta’s Llama 3 blog post, which made the case for open weights, is another reference point. He hopes Meta keeps pursuing that path.

The discussion also covers how pre-training has co-evolved with inference and post-training. Long-context generation has made inference scaling so important that architecture choices, such as sparse attention making 100,000-token generation 40 percent faster, now yield large returns. Pre-training, mid-training, and post-training form a more complicated cycle, and getting that off the ground takes people with the right experience. Lambert’s own experience at AI2 taught him that post-training cycle time matters a great deal. He wants to fix that bottleneck, since the bar for open-source post-training is low and modest capital can produce large compounding gains.

Harnesses, Agent Swarms, and Where the Gap Goes

The final thread concerns harnesses. Closed labs train their models to work well in their own agent harnesses, which can increase the mystique. Lambert initially expected closed labs to behave like Apple, with integrated model and harness design, but the market pushed model companies to work across many harnesses. Now agent swarms are emerging, and OpenAI and Anthropic will likely design for them first, creating a coordination system rather than a traditional harness. Lambert expects the familiar cycle: frontier labs push new approaches, open models follow once the shape of the approach is clear, and then post-train on similar meta-harnesses. He doubts the labs can keep these methods fully secret, noting that Silicon Valley talks and ideas travel quickly, and that serving efficiency is where labs can truly win. The labs can serve cheap intelligence per watt at scale, and that is good business, though coding is currently the most profitable domain and the broader knowledge-work economy is still wide open.

Lambert ends with a sober view of the scaling picture. Exponential spending on compute yields roughly linear gains in performance, which the labs are sustaining. Hard breakthroughs could come from a different architecture or a new kind of intelligence, such as an AI discovering something that changes the transformer, and that would merit updating. Mathematics is the field he thinks best suits this shape of intelligence. Without such a shift, he is comfortable with the labs continuing to scale, but he acknowledges that no one knows exactly what comes next.

He closes on preparedness. The gap that worries him most is not RSI but the thin preparation for cross-public cybersecurity and similar risks. Risks are already here and diffusing, and the open-model trajectory will bring similar problems. Society needs more eyes and more science so it feels informed, and not fearing a fast takeoff does not mean ignoring risks.

• Tags:
• Synthetic Data
• Post-Training
• Recursive Self-Improvement
• Open Models
• Distillation
• RL Environments
• Agent Harnesses
• Frontier Labs
• Compute Constraints
• Open Research