The State of Frontier Post-Training Recipes | Conversation with Finbarr Timbers

The conversation between Nathan Lambert and Finbarr Timbers delves into the evolving landscape of frontier post-training recipes for large language models, reflecting on Finbarr’s recent departure from AI2 and the challenges and innovations encountered in building models like Olmo.

Reflections on Olmo’s Post-Training Journey

Finbarr’s last day at AI2 prompted a reflection on the development of Olmo 3, particularly its post-training phase. While achieving a strong reasoning model was a significant accomplishment, the complexity pushed AI2’s organizational capacity to its limits. Modern post-training, it’s argued, is less about purely technical challenges and more about “wrangling compute, data into a workstream,” which often translates to “wrangling an org chart.” Olmo 3’s recipe was relatively simple, characterized by a three-stage SFT-DPO-RL approach, akin to Tulu 3. However, this simplicity placed it at a “fork in the road” compared to the increasingly sophisticated methods employed by frontier labs, especially for reasoning and tool-use models.

The discussion highlighted the speed of model development, noting that while Olmo 3 was released in late 2025, about nine months after DeepSeek R1, this turnaround time was considered acceptable in some contexts. However, the core issue wasn’t speed but the approach: Olmo ported reasoning into an existing, simpler recipe rather than rebuilding it from scratch, which limited its “ceiling” compared to newer, more prescriptive recipes.

The Evolution of Frontier Post-Training Recipes

The conversation provides a historical overview of canonical post-training recipes, identifying key shifts:

• InstructGPT Era (Pre-ChatGPT): Characterized by a three-step process: Supervised Fine-Tuning (SFT) using human demonstration data, training a Reward Model (RM) from human preference rankings, and then Reinforcement Learning (RL). This approach, also seen in models like Llama 2 and Llama 3 (with added tricks), has largely been “phased out” for its original components.

  • Key Shift: Less reliance on human demonstration data for SFT (synthetic data plays a much larger role), human preference data is still present but reward models are no longer the key RL target. DPO (Direct Preference Optimization), while used in Olmo 3 to refine probability distributions from strong open-weight model outputs (like DeepSeek), is increasingly absent in frontier recipes due to their more refined, industrial processes.

• DeepSeek R1 (Early 2025): Marked a significant shift towards reasoning-focused models and larger, more iterative RL stages. DeepSeek’s recipe involved:

  1. Starting with a base model.
  2. An initial RL0 stage.
  3. Sampling from RL0 to filter prompts, used as SFT for the next model version.
  4. Repeated sampling to train multiple RL versions, refining reasoning behavior.
  5. A final pipeline combining reasoning and non-reasoning SFT into a larger RL run (using GRPO). This process emphasized the use of synthetic data as a primary input, serving SFT as a “cold start” for RL.

• DeepSeek V3.1 (Mid-2025): Merged “thinking” and “non-thinking” into a single model, a complex but necessary change for serving perspective, anticipating that all models will eventually be reasoning models due to efficiency gains.

• DeepSeek V3.2 (Late 2025): Introduced significant changes, including “expert creation with separate mini recipes” and incorporating these into the R1 data process for SFT and a large RL run. This represented a major increase in complexity and industrial scale.

• DeepSeek V4 (April 2026): Further expanded with more experts and a new loss function for Multi-Teacher On-Policy Distillation (MOPD). This exemplifies the industry’s arc towards scaled RL and integrating more domains without operational grinding.

Multi-Teacher On-Policy Distillation (MOPD)

The “2026 style recipes” are characterized by MOPD, first coined in the MIMO Flash v2 paper.

• Core Concept: Within an RL framework, the model being trained (the “general model”) samples its own trajectories. These trajectories are then routed to various pre-trained “expert models” (teachers), each specializing in a domain (e.g., search, code, math, reasoning, safety). The general model is then trained with a distillation KL loss to match the tokens of the relevant expert for that trajectory. This form of supervision, often combined with other RL losses (like verifiable rewards), proves highly effective.
• Expert Training: Experts themselves are trained using complex, domain-specific recipes, often akin to the DeepSeek R1 style, to be highly proficient in their narrow task.
• Challenges (Nvidia Nebatron 3 Ultra): Nvidia’s Nebatron 3 Ultra, a strong US-made model, highlighted difficulties in MOPD. A key finding was that “teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward on policy distillation merge, resulting in suboptimal performance.” This “distribution mismatch” between student-generated trajectories and teacher distributions necessitates “cross-teacher alignment” and careful integration, potentially leading to organizational complexities.
• Alternative Approaches (Microsoft MAI Thinking): Not all labs immediately adopt full MOPD. Microsoft’s MAI “Thinking” model used a “conservative recipe” with three more general experts combined via SFT and a longer RL run, resembling DeepSeek R1 more closely. The hypothesis is that they might later introduce finer-grained teachers and potentially switch to MOPD.
• Chinese vs. US Labs - Technical Nuances: Chinese labs are perceived as more willing to share “nitty-gritty tech details” and show a trend towards sparse attention, while US labs (Nvidia, AI2) lean towards hybrid attention (e.g., Mamba).

Open-Ended Discussions and Future Directions

The conversation also touched on broader trends and open questions:

• Environments: The high cost of specialized environments (e.g., $100k+ per environment for sophisticated web apps) poses a challenge. However, there’s optimism for open environments, where small teams could build good evaluation domains, and for companies to eventually release their own environments for marketing or to integrate agents (e.g., DoorDash releasing its environment for LLMs).
• Tinker-Style APIs: These APIs offer RL frameworks runnable on GPUs, abstracting complexity and providing a potential avenue for academic research and smaller labs. While beneficial for accessibility and potentially more sustainable economics than large GPU purchases, their business model (beyond selling raw compute) is still evolving, potentially feeding into inference services.
• Resource Allocation (Pre-training vs. Post-training): A critical and often opaque decision for labs is how to allocate compute and human resources between pre-training and post-training. This choice reflects where labs believe the most significant progress and ROI lie, shaping the direction of frontier AI development.

Career Advice in the LLM Race

Finally, the speakers offered career advice for those in the competitive LLM space:

• Focus on Long-Term Value: Prioritize work that is “generally valuable” and pushes scientific boundaries, rather than short-term “exploitation.”
• Seek Interesting Work and Learning: Organizations that pay well often also offer intellectually stimulating work and ample resources for learning and growth.
• Conviction Over Trends: Success stories (like DeepMind’s poker group or AI2’s long-tenured NLP researchers) often stem from individuals with “high conviction” in their chosen area, rather than simply following the latest fad. This leads to deeper expertise and significant impact.
• Embrace Diversity of Approaches: For junior researchers, there’s an opportunity to pursue convictions outside the established labs, fostering a more diverse set of approaches and preventing excessive convergence on similar methodologies.

• Tags:
• Synthetic Data
• Reinforcement Learning
• Model Distillation
• AI Models
• RLHF
• Post-Training
• AI Careers
• DeepSeek
• AI Research
• SFT
• DPO
• Olmo
• LLM Recipes
• Multi-Teacher On-Policy Distillation
• MOPD
• Supervised Fine-Tuning
• Nvidia Nebatron
• Organizational Capacity