Build Hour: Reinforcement Fine-Tuning

A deep dive into Reinforcement Fine-Tuning (RFT), covering how to set up tasks, design effective graders, and run efficient training loops to improve model reasoning, based on a live demonstration from OpenAI's Build Hours.

September 3, 2025 5 min read

Build Hour: Reinforcement Fine-Tuning — source: https://www.youtube.com/watch?v=YWLOo_fc5oA

Reinforcement Fine-Tuning (RFT) is a powerful technique for model customization that focuses on improving how a model reasons, rather than just what it knows. While context engineering techniques like Retrieval-Augmented Generation (RAG) are best for knowledge gaps, fine-tuning is appropriate when a model possesses the necessary facts but struggles to apply them correctly. RFT is a significant investment and should be considered only after exhausting prompt engineering and RAG.

Unlike other fine-tuning methods, RFT is uniquely suited for enhancing reasoning, making it a forward-looking approach. It is also highly data-efficient, often requiring only tens to hundreds of examples to show meaningful improvement.

The RFT Workflow: A Practical Guide

The process was demonstrated using a legal text classification task, where documents must be assigned to one or more of 21 EuroVoc categories.

1. Dataset Selection and Preparation

The quality of your data is paramount in RFT. The initial dataset, sourced from Hugging Face, contained thousands of samples but was highly imbalanced across categories.

Data Transformation: The original dataset used numeric IDs for categories, which lack semantic meaning for the model. The first step was to map these IDs to their human-readable names (e.g., “Trade”, “Finance”) to enable the model to reason about the concepts.
Balanced Sampling: An imbalanced training set can lead to the model “reward hacking” by predicting only the most frequent classes. To counteract this, a balanced sampling strategy was employed for the training set of 100 examples, ensuring a more uniform distribution of classes. This encourages the model to develop generalizable reasoning rather than exploiting statistical quirks in the data. For the validation set, however, it’s often useful to maintain the original, real-world distribution to get a practical measure of production performance.

2. Grader and Prompt Engineering

RFT operates not with labeled answers, but with graders—functions that provide a score for the model’s output. For a classification task, the key metrics are precision and recall.

Precision: Of the labels the model predicted, how many were correct?
Recall: Of the actual correct labels, how many did the model find?

These are combined into an F1 score, which provides the single, continuous reward signal required by the RFT system.

The Python code for the graders must be robust, handling edge cases where the model might not return the expected output format.

# Example Precision Grader
def precision_grader(history, **kwargs):
    """
    Calculates precision between the reference and completion lists of labels.
    """
    completion_labels = history[-1]["completion_labels"]
    reference_labels = history[-1]["reference_labels"]
    if not isinstance(completion_labels, list) or not isinstance(reference_labels, list):
        return 0.0
    if not completion_labels:
        return 1.0 if not reference_labels else 0.0
    
    true_positives = sum(1 for label in completion_labels if label in reference_labels)
    return true_positives / len(completion_labels)

The prompt is another critical component. A simple, clear prompt that defines the task, specifies the output format, lists the possible classes, and provides guidelines often outperforms complex prompts with few-shot examples. A shorter prompt is also faster and more cost-effective during training and inference.

3. Establishing a Baseline and The Variance Study

Before starting an RFT job, it’s crucial to establish a baseline using the Evals platform. This involves running the validation set against the base model (e.g., gpt-4o-mini) to understand its initial performance.

A key analysis at this stage is the variance study. By running the evaluation multiple times (e.g., nine times), you can observe the performance variance for each sample.

A wide variance, where the model sometimes scores high and sometimes low on the same sample, indicates headroom for improvement.
This variance shows that better reasoning paths exist within the model’s potential, and RFT can learn to consistently follow these higher-scoring paths. If the model consistently scores poorly with low variance, RFT is unlikely to succeed.

4. Training and Analyzing the RFT Job

With a prepared dataset, robust graders, and a validated baseline, the RFT job can be initiated. The platform provides a detailed dashboard for monitoring the training process.

Reward Curves: The primary chart tracks the reward (F1 score) over training steps for both the training and validation sets. The goal is to see both curves trending “up and to the right.” If the training reward increases while the validation reward stays flat, the model is overfitting.
Component Metrics: The dashboard also breaks down the reward into its components (e.g., precision and recall). This allows for nuanced analysis. For instance, you might observe that at a certain checkpoint, the model achieves very high precision at the cost of recall. This insight helps in selecting the checkpoint that best aligns with the desired behavior for your application.
Reasoning Tokens: The training process also tracks the number of chain-of-thought tokens. It’s common to see the model experiment with longer reasoning paths before converging on a more efficient strategy. This has direct implications for latency and cost.

The final fine-tuned model showed significant improvement, lifting both precision and recall higher than the gpt-4o-mini base model and even surpassing gpt-4-turbo on this specific task.

Real-World Application: Accordance

David, the CEO of Accordance, shared how they use RFT for their AI platform for tax and accounting professionals.

Task Selection: They chose a tax optimization task, which is highly objective and requires complex reasoning—a perfect fit for RFT.
Grader Design: A simple “right or wrong” grader is ineffective because it encourages guessing. Accordance designed a continuous, multi-part grader that rewarded the model based on how much it deviated from the optimal tax saving, while also penalizing it for non-viable strategies. This prevents reward hacking and guides the model toward correct reasoning pathways.
Results: Using a small, high-quality dataset and intelligent grader design, Accordance achieved a 40% performance improvement on the TaxBench industry benchmark.