Post-training best-in-class models in 2025
An expert overview of post-training techniques for language models, covering the entire workflow from data generation and curation to advanced algorithms like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning (RL), along with practical advice on evaluation and iteration.
Post-training is the crucial process of transforming a pre-trained base model, which can only perform token completion, into a sophisticated model capable of following instructions and answering questions. This process is an iterative cycle of data curation, training, and evaluation, essential for creating specialized and high-performing models.
Supervised Fine-Tuning (SFT): The Foundation
The first step in post-training is Supervised Fine-Tuning (SFT). This involves training the base model on a large, high-quality dataset of instruction-answer pairs, often exceeding one million samples for general-purpose models.
Data Quality and Structure
The quality of the SFT dataset is paramount. A good dataset must be:
- Accurate: The answers must be factually correct.
- Diverse: It should cover a wide range of topics and tasks.
- Complex: The tasks should be challenging enough to facilitate model learning.
The typical data structure includes an optional system prompt, a user instruction, and the expected model output. During training, the loss is calculated only on the model’s generated output, making the quality of the provided answers critically important. A common data generation pipeline involves using a powerful LLM to generate responses to seed prompts with specific constraints, followed by automated checks, filtering, and decontamination to prevent training on test data.
SFT Techniques and Parameters
- Full Fine-Tuning: Updates all model parameters, maximizing potential quality but requiring significant computational resources.
- Parameter-Efficient Fine-Tuning (PEFT):
- LoRA (Low-Rank Adaptation): Freezes the base model’s weights and introduces small, trainable matrices (adapters). This drastically reduces the number of trainable parameters (e.g., to 0.1%), saving VRAM and speeding up training.
- QLoRA (Quantized LoRA): Further reduces memory requirements by loading a quantized (e.g., 4-bit) version of the model before applying LoRA. This is a trade-off, as it can lead to a degradation in quality compared to standard LoRA.
The most critical hyperparameter to tune is the learning rate. An excessively high learning rate can cause “loss spikes,” leading to a collapse in model performance. Monitoring the training loss for a smooth, descending curve is a key indicator of a successful run.
Preference Alignment with DPO
Direct Preference Optimization (DPO) is a powerful technique for aligning a model’s behavior and style with human preferences. It moves beyond simple instruction-following to refine the nuances of the model’s responses.
DPO uses a preference dataset composed of prompts, “chosen” (preferred) answers, and “rejected” (less preferred) answers. The training objective is contrastive: it increases the likelihood of the model generating responses similar to the chosen examples while decreasing the likelihood of generating those similar to the rejected ones.
A key hyperparameter in DPO is beta, which controls how closely the model must adhere to the reference model. A low beta allows for more exploration, while a high beta keeps the model’s behavior constrained.
DPO is highly effective at creating models that humans prefer, as measured by metrics like the Chatbot Arena Elo score. However, it’s important to note that human preference is often weakly correlated with performance on academic benchmarks for tasks like math or reasoning.
Advanced Reasoning with Reinforcement Learning (RL)
For complex reasoning tasks like math and coding, Reinforcement Learning (RL) offers a powerful training paradigm. A popular approach, used for models like DeepSeek, involves a multi-stage process.
- SFT Warm-up: The model is first fine-tuned on a specialized dataset where each answer is preceded by a “reasoning trace” or chain-of-thought. This teaches the model to structure its thinking process before providing a final answer.
- RL Training: The model is then trained using RL, where a reward function provides a signal based on the correctness of the final answer. The model is rewarded for correct answers and penalized for incorrect ones.
This process allows for multi-objective optimization. For instance, a 350M parameter math model was trained with RL not only to improve its mathematical ability but also to compress its reasoning traces. This resulted in a model that achieved similar performance with significantly shorter, more efficient outputs (e.g., 4,000 tokens instead of 32,000), improving inference speed.
Unlike typical training, the loss during RL (specifically, the KL divergence) is expected to rise, indicating that the model is learning and diverging from its baseline policy.
The Iterative Cycle of Post-Training
Achieving a state-of-the-art model is not a one-shot process but an iterative cycle.
- Data: The cycle begins with meticulous data collection, generation, and filtering. This is the most critical stage.
- Train: Select and apply the appropriate training algorithms (SFT, DPO, RL, or even model merging, which averages the weights of different checkpoints).
- Evaluate: Use a suite of relevant benchmarks to assess the model’s performance.
- Refine: The evaluation results provide a feedback signal, highlighting areas for improvement (e.g., “the model is weak at math”). This informs the next cycle of data curation and training.