How We Built a Leading Reasoning Model (Olmo 3)

Building a frontier model like Olmo 3 Think involves a multi-stage process that considers every part of the stack, from pre-training and architecture to post-training and evaluation. The primary motivation for creating another “thinking” model was to provide the research community with a clean, fully open base model for Reinforcement Learning (RL) research, free from the potential data contamination issues observed in other models.

The Problem of Spurious Rewards

Recent research has shown that the open-source reasoning model landscape can be messy. For instance, some models show performance improvements when trained with RL on random rewards. A leading hypothesis for this phenomenon is that the base models were pre-trained on data very similar to common evaluation benchmarks. When RL is applied, even with spurious rewards, the optimization process may simply amplify these high-probability, pre-learned behaviors rather than fostering genuine reasoning. This underscores the need for a transparently trained base model to ensure that RL research results are trustworthy.

Pre-training and Mid-training Stages

The journey begins long before post-training, with crucial decisions made during the pre-training and mid-training phases.

Architecture’s Impact on Post-training

Architecture choices made during pre-training have significant downstream consequences, especially for RL-heavy post-training that involves substantial inference. A key example is the absence of Grouped-Query Attention (GQA) in the Olmo 7B model. GQA drastically reduces the memory footprint of attention heads. Its absence meant that the 7B model required significantly more GPUs for inference during RL compared to the much larger 32B model which included GQA. This highlights a critical lesson: architectural efficiency is paramount for the feasibility and speed of post-training experimentation.

Data Selection and Long Context

The data mix for Olmo 3 was shifted to more heavily prioritize reasoning, math, and code. The selection process was methodical, involving training hundreds of small models on different data sub-sources and fitting a regression to predict the optimal mix for downstream performance.

A prerequisite for any modern reasoning model is long-context capability. Reasoning through complex problems can require tens of thousands of tokens. For Olmo, achieving effective long context was found to be highly dependent on architecture. While more compute can sometimes compensate for a suboptimal architecture, having a strong architectural foundation is the most efficient path, especially for compute-limited teams.

The Post-training Recipe

The post-training for Olmo 3 Think follows a three-stage recipe: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning from Verifiable Rewards (RLVR). This approach is framed as a “distillation recipe,” ideal for instilling the capabilities of larger teacher models into smaller, more efficient ones.

1. Supervised Fine-Tuning (SFT)

The foundation of the recipe is a high-quality SFT dataset. This was created by:

• Using strong teacher models like DeepSeek Coder V2 and Qwen 2 32B.
• Extending existing datasets, such as taking the 16k-context Open-Orca dataset and extending its truncated samples to 32k.
• Aggressive filtering to improve data quality and model stability. This involved removing non-commercial data, multimodal prompts, samples with formatting errors (e.g., missing think tags), and identity-confused responses (e.g., “I am DeepSeek”). A crucial step was filtering out samples where the model enters repetitive loops, a behavior observed in about 1 in 1,000 to 10,000 samples from teacher models.

2. Direct Preference Optimization (DPO)

Despite being colloquially frowned upon by some, DPO provided significant, low-effort performance gains. The approach was inspired by the “delta learning hypothesis,” which posits that the contrast between the chosen and rejected responses is more important than the absolute quality of the chosen response.

To create a strong learning signal, preference pairs were generated using two different Qwen models: the powerful 32B reasoning model for the “chosen” responses and the weaker 0.5B model for the “rejected” ones. This created a consistent and clear delta for the model to learn from. DPO alone added about two points on average to evaluation benchmarks, a gain comparable to the much more complex RL stage, and the benefits appeared to stack with subsequent RL training.

3. Scaling Reinforcement Learning (RL)

The most complex and intellectually interesting part of the process was building the infrastructure for RLVR on long-context tasks. This was a massive engineering effort.

The RLVR Infrastructure

The core challenge is efficiently managing a learner (which performs the gradient updates) and many actors (which generate completions). The goal is to minimize idle GPU time and maximize throughput. Several key techniques were implemented:

1. Continuous Generation: Actors generate data continuously, which is then used by the learner. This makes the process slightly “off-policy” (data is from a slightly older policy version), but the performance impact is minimal compared to the efficiency gain.
2. Importance Sampling: To handle the numerical mismatch in log probabilities between the generation framework (vLLM) and the training framework (Hugging Face Transformers), importance sampling is used to re-weight the advantage, correcting for the probability delta.
3. In-flight Updates: Generating very long sequences creates a bottleneck, as the system must wait for the slowest generation to finish. To solve this, the model’s weights are updated while a long generation is still in progress. This sounds numerically strange but works well in practice and provides up to a 4x speedup.
4. Active Sampling: In RL, if all generated answers for a prompt are incorrect (or all are correct), the resulting gradient is zero, and the sample is filtered out. This leads to variable batch sizes and training instability. Active sampling addresses this by creating an intermediate buffer. The system waits to form a batch until it has collected enough samples that are guaranteed to have a non-zero gradient, ensuring a full, effective batch for every training step. This dramatically stabilizes the RL loss curve.

The Nuances of Evaluation

A robust evaluation suite is critical for trusting the development process. Building the Olmo 3 suite involved moving away from multiple-choice questions to more challenging Chain-of-Thought (CoT) formats with non-zero temperature, which better reflect real-world reasoning.

Reflections on Evaluation

• High Variance: Reasoning models evaluated with sampling have high variance. A reported gain of 0.5 points on a benchmark could easily be noise. For some benchmarks like AIME, averaging over many runs (e.g., avg@32) is necessary to get a stable signal.
• Averages are Misleading: Averaging scores across a suite of benchmarks can hide a model’s specific strengths and weaknesses. It’s more informative to look at individual scores to determine if a model is right for a specific task.
• Cost: Evaluation is extremely expensive, accounting for around 20% of the total compute budget for post-training. A single evaluation run can take hours and hundreds of GPUs.
• “Benchmaxing” is Inevitable: When you have a target set of evaluations for development, you will inevitably optimize for them. The key is to do so in a scientifically rigorous way and test for generalization on unseen evaluation sets.
• We Need More Evals: The community is good at hill-climbing on existing benchmarks. To build truly robust and general models, a much larger and more diverse set of evaluations is needed, especially for qualitative aspects like chat.

Conclusion and Future Work

The Olmo 3 Think models provide a solid, fully open platform for RL and agentic research. The entire process, from data to evaluation, has been made transparent. Future work will focus on scaling RL to even larger models, tackling more complex agentic behaviors like tool use and search, and exploring Mixture of Experts (MoE) architectures.

• Tags:
• LLM
• Evaluation
• Reinforcement Learning
• Feature
• Long Context
• Model Training
• SFT
• DPO
• Olmo