Debating RSI, the US-China Gap, and Jaggedness of Frontier AI

This episode features a discussion between Jean-Stanislas “JS” Denain of Epoch AI and Nathan Lambert (NL), delving into the current state and future trajectory of AI, particularly concerning Recursive Self-Improvement (RSI), the US-China AI gap, and model safety. They both express significant uncertainty about the future, presenting their observations from today.

Predictions for RSI and Capabilities

NL initiates by asking what internal measurements from OpenAI and Anthropic are driving urgent RSI proclamations. JS believes public evidence of accelerating AI progress is already substantial enough to warrant concern, rather than a single secret data point. He suggests internal KPIs, like “compute multipliers” in pre-training, could serve as early warning signs.

On the link between RSI and existential risk, JS, while uncertain, is “very worried” and identifies AI capabilities as the “principal component of disagreement” in these discussions. He describes a “capabilities maximalist” scenario where AI automates its own research, leading to massive robotics progress, an industrial explosion, AI-managed factories, a self-sustaining economy, and rapid scientific advancements (e.g., the I2027 scenario).

NL expresses skepticism regarding a 10x net productivity increase for AI researchers, despite agreeing that experimentation cycles could speed up by 10x. He differentiates between junior PhDs becoming 10x productive and the overall progress of an advisor’s research agenda, noting that company-wide output faces bottlenecks like compute. NL also points out that the massive influx of inference compute makes it difficult to disentangle AI research speed-up from mere compute spending. JS acknowledges OpenAI’s CoPilot/Codex spending surge for engineering tasks but notes less uplift for high-level strategic decision-making.

NL anticipates a fundamental transformation in science, moving into a rapid AI-driven era. He identifies pre-training scaling laws and RL environments for knowledge work as clear drivers of new capabilities. He’s more optimistic about scaling efficiency than scaling “raw intelligence” for complex, open-ended problems like curing cancer, although he views mathematics as a unique, falsifiable exception. JS concurs that efficiency scaling is a hill-climbing task with fast trends.

They discuss the Epoch Compute Index (ECI), with JS finding a 5x acceleration in ECI slope for frontier models likely. NL agrees but warns of potential saturation as measurable “hill-climbing” aspects are optimized. He highlights the difficulty and cost of building evaluations for fundamental breakthroughs, contrasting it with more measurable tasks like coding or ML research.

Robotics in AI Acceleration

NL posits that robotics trends will more closely resemble the slow, complex rollout of self-driving cars than the rapid scaling of LLMs. He argues against the singularity-driven assumption of LLM-like robotics growth, citing the difficulty of building things in the real world and the slower progress of visual and action models compared to text-based LLMs. He notes that while Amazon builds new factories to be robot-first, this requires new infrastructure built by robots, a challenge for re-industrialization.

JS counters that sufficient robot capabilities could lead to massive buildouts due to strong financial incentives, citing data centers as an example of rapid expansion when demand is high. He questions whether extreme dexterity is essential for an industrial explosion, or if sufficient capabilities are enough. NL emphasizes that robots will excel in constrained, repetitive environments (like manufacturing) but cohabiting with humans on the streets is a much longer-term prospect.

The US-China AI Gap and Distillation

JS estimates a 6-8 month lag for top Chinese labs (public models) compared to OpenAI/Anthropic. NL suggests specific Chinese models like Kimi K3 and GLM 5.2 might be closer (2-4 months). JS acknowledges this, noting that while overfitting to benchmarks might slightly narrow the perceived gap, Epoch’s internal analysis hasn’t found it statistically significant. He suspects some un-benchmarked capabilities still indicate a wider gap.

The discussion then turns to distillation’s role. JS has updated his view, now seeing it as a “really big factor,” estimating that without it, the gap would increase to 8-9 months. He points to the “stolen thoughts” paper and “cloud routers” (providing realistic usage distributions) as evidence. NL acknowledges router utility but questions its impact on benchmark metrics, while JS adds that mid-training and Reinforcement Learning from Human Feedback (RLHF) on good trajectories are key.

NL maintains that RL is primary, but notes the current framework involves a repeated cycle of mid-training, Supervised Fine-Tuning (SFT), and RL. He doubts distillation’s scale for mid-training would be sufficient to explain the perceived small gap with Chinese models, suggesting the gap would remain similar even without it due to abundant “low-hanging fruit.” He also notes that Chinese labs, despite having less compute/capital, might compensate by being better at organizing talent for “mundane hill climbing data work” and by waiting to purchase RL environments from frontier labs at a lower cost, boosting their capital efficiency. NL believes frontier labs lead in “open-ended innovations,” while Chinese labs focus on catching up, facilitated by rapid personnel turnover that leaks architectural innovations.

Chinese Job Postings and Data Market

Referring to a post by Cheryl Woo, JS discusses how analyzing Chinese job postings offers insights, like companies building their own data centers or revealing product strategies (e.g., JIPU’s focus on B2B sales, Minimax’s international expansion). NL asks if Epoch could automate this. JS finds LLMs useful for classifying job types and tracking hiring trends but less effective for identifying nuanced “interesting” takeaways. They discuss the difficulty of tracking the secretive data market, noting Chinese companies tend to do more in-housing, including for data, and extensively hire interns.

Open vs. Closed Model Safety

NL brings up the recent news of three individuals using public cloud models to hack OpenAI, questioning model safeguards. JS notes this raises questions about both model safeguards and company security. He contrasts open models (less powerful, worse safeguards) with closed models (powerful, better safeguards), finding it unclear which poses a greater risk. He references the “stolen thoughts” paper as evidence that safeguards haven’t always been sufficient. NL views this as a serving issue, suggesting models are “permanently leaky,” and that a strong AI could likely jailbreak another. He believes the current lack of oversight and strong classifiers in open models is a bigger concern than malicious fine-tuning, as not enough people have the motivation or competence for the latter.

JS stresses the importance of “actual threat modeling” to identify the most likely bad actors and determine if open or closed models are riskier. He concludes that for now, the biggest safeguard is that “not that many people want to do horrible things.” NL finds some reassurance in the fact that “all hell is not breaking loose” despite current inadequate safeguards and open models, though he expects “Hugging Face style things” (uncontrolled agents) to become more common.

Epoch AI and Post-Training Recipes

JS explains Epoch AI’s project selection is curiosity-driven, focusing on important trends and addressing misconceptions, filling a gap for public resources on AI trends. He notes that the accessibility of information varies; data centers are becoming easier to track, while model training compute is harder due to less public research. Data remains difficult to quantify.

NL asks for insights into frontier labs’ post-training recipes. JS speculates on a complex process: specialized sub-teams curating data, optimizing RL environments, and finding hyperparameters for their domains, using high-quality evaluations. These teams then “bid” to contribute their data and hyperparameters to a final, large RL training run. NL suggests that the diversity in open-weight models’ post-training (e.g., multi-tier on-policy distillation vs. sequential RL) is significant. He believes knowing if labs are using a simpler “mid-training then big RL” approach (faster progress) versus a complex, multi-stage, expert-driven approach would clarify potential bottlenecks in progress. JS’s vague sense is a shift away from multi-expert/MOPD towards more unified large runs.

• Tags:
• LLMs
• Robotics
• Threat Modeling
• AI Capabilities
• Open Models
• Distillation
• Model Safety
• Recursive Self-Improvement (RSI)
• Existential Risk (X-Risk)
• US-China AI Gap
• Epoch Compute Index (ECI)
• Closed Models
• Epoch AI
• Post-Training Recipes
• ML Research