913: LLM Pre-Training and Post-Training 101 — with Julien Launay

The process of creating Large Language Models (LLMs) has historically been divided into two main phases: pre-training and post-training. While these steps are beginning to blend, they serve distinct purposes.

The Two Phases of LLM Creation

1. Pre-training: This is the foundational, compute-intensive phase where a model learns from a massive corpus of data scraped from the internet, books, and papers. The objective is simple and scalable: predict the next word (or “token”). However, a purely pre-trained model is often “unwieldy” and not inherently interactive. If you ask it a question, it might just generate more similar questions, as this pattern is equally likely in its training data.

2. Post-training: This phase sharpens the model for its intended use case, such as being a helpful chat assistant. While this can be done by continuing pre-training on specialized data (e.g., chat transcripts), the most significant breakthroughs have come from Reinforcement Learning (RL). Instead of learning from explicit examples, the model learns from feedback—a signal indicating whether its output was good or bad. This process is becoming increasingly crucial, with some frontier models like Google’s Gemini and xAI’s Grok dedicating nearly as many resources to post-training as to pre-training.

The Spectrum of Reinforcement Learning Feedback

The feedback used in post-training can come from several sources, each with its own advantages and scaling properties:

• RLHF (Reinforcement Learning from Human Feedback): This was the initial breakthrough, famously used in ChatGPT. Humans provide feedback, often through a simple thumbs-up/thumbs-down mechanism. However, this method is limited by the scalability and cost of human annotation. Human annotators also suffer from low inter-annotator agreement, introducing significant noise.

• RLEF (Reinforcement Learning from Execution Feedback): Also known as RLVF (from Verifiable Feedback), this involves tasks where success is objectively measurable. Examples include solving a math problem, passing a coding test, or achieving a high score in a game. This is highly scalable as the feedback is generated automatically by an environment. It’s a return to the classic RL paradigm of AlphaGo and Atari games.

• RLAIF (Reinforcement Learning from AI Feedback): This highly scalable method uses another AI model (a “judge” model) to evaluate the primary model’s output. It’s surprisingly effective, as verification is often an easier task than generation. The quality of AI-generated feedback can be comparable to or even exceed that of human annotators, especially when you consider the noise in human-labeled data. This is a cornerstone of the “synthetic data” trend, allowing for massive scaling of the training process. For example, from just 70-80 human-annotated samples, it’s possible to generate 80,000 synthetic conversations, leading to nearly 10 million data points for the model to learn from during RL.

Supervised Fine-Tuning vs. Reinforcement Learning

A key distinction lies in how the model learns.

• Supervised Fine-Tuning (SFT) is analogous to memorizing the solution to an exercise. You show the model an explicit example, and it learns to replicate it. If the problem changes slightly, the model may fail because it hasn’t generalized.
• Reinforcement Learning is more like having a teacher correct your work as you attempt an exercise. The model generates its own samples (“trial and error”) and is nudged in the right direction by a reward signal. This “online” nature, where the model learns from data it just produced, helps it explore and shape its capabilities more effectively, leading to better generalization. It learns from a much wider range of signals beyond simple imitation.

Adaptive ML: Providing “RLOps” for the Enterprise

Reinforcement Learning, while powerful, is notoriously difficult to implement due to its complex engineering requirements. This is the problem Adaptive ML solves.

• The Challenge: RL pipelines require blending model inference and training, which is complex. They must interact with multiple external systems simultaneously, such as several AI judge models, SQL databases for text-to-SQL tasks, code compilers, or even virtual machines for tool-use tasks.
• The Solution: Adaptive ML provides the “RLOps” (Reinforcement Learning Operations) tooling that abstracts away this complexity. It allows data science teams to define their logic—the judges, environments, and data pipelines—in Python, and the platform handles the distributed execution on a cluster.
• The Goal: The aim is to put powerful RL methods into the hands of every data scientist, enabling them to tune smaller, more efficient models to achieve frontier performance on specific tasks. This helps enterprises optimize for either cost or performance. Adaptive ML operates as a layer on top of powerful open-source foundation models like Llama or Mistral, sharpening them for a specific use case.

The Future: Scaling, Hardware, and AGI

• Beyond the Data Bottleneck: While we may be approaching the limits of available human-generated text data for pre-training, the future of scaling lies in post-training. Models can learn from experience by interacting with simulated or real-world environments. This source of data is practically infinite. The next bottleneck is enabling models to run experiments at scale in the real world, for instance, by controlling automated wet labs for biology or materials science.

• The Hardware Question: Current silicon-based GPUs (like NVIDIA’s) are the most complex objects ever created by humanity. While they are energy-intensive, creating a new hardware paradigm (e.g., photonic or quantum computing) from scratch is a monumental challenge. The current hardware is likely sufficient to bootstrap the creation of Artificial General Intelligence (AGI). These future AGI systems will then help us design and build the next generation of computing hardware.

• The Path to AGI: We will likely build superintelligent systems within the next 5-10 years. However, the arrival will be “messy” and “spiky.” The AGI will be superhuman in some domains (like mathematics, which can be explored entirely within a computational box) but will remain bottlenecked in others that require physical experimentation (like medicine). This will lead to an era of uneven, but astonishing, progress.

• Tags:
• LLM
• Synthetic Data
• Reinforcement Learning
• Feature
• RLHF
• AGI
• RLAIF
• Adaptive ML
• RLOps