Tokenless

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

A practical guide to the engineering principles and trade-offs involved in training a small language model from scratch on a local machine, based on a workshop by Angelos Perivolaropoulos from ElevenLabs.

5 min read
Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs
source: https://www.youtube.com/watch?v=UsB70Tf5zcE

The Four Pillars of Training an LLM from Scratch

Training a large language model (LLM) from scratch is often seen as an endeavor exclusive to large labs with massive compute resources. However, the fundamental principles can be applied on a much smaller scale, even on a local machine. The process revolves around four key building blocks: the tokenizer, the model architecture, the training loop, and the inference logic.

1. The Tokenizer: The Language of the Model

The first and one of the most critical decisions is choosing a tokenizer, which converts raw text into numerical representations (vectors) that the model can process. For this hands-on project, a character-level tokenizer is used.

  • Simplicity and Small Vocabulary: This approach is ideal for small-scale training because the vocabulary is tiny—only 65 unique characters in the Shakespeare dataset. This means the model has fewer possible combinations to learn (e.g., 65 * 65 = 4,225 possible bigrams), making it converge faster on a small dataset.
  • The Trade-off: The major drawback is that character-level tokenizers don’t scale well. The model struggles to understand the relationship between semantically linked but distant characters (e.g., the characters in “sky” and “blue”). It’s computationally expensive and less effective for building powerful, general-purpose models.
  • Advanced Alternative: In contrast, large-scale models use more sophisticated tokenizers like Byte Pair Encoding (BPE). BPE is trained on the entire dataset to find and merge common character patterns into single tokens (e.g., “ing”, “the”, “for”). This creates a much larger vocabulary but allows the model to learn meaningful relationships between word parts and concepts more effectively.

The tokenizer’s vocabulary size directly impacts the model’s embedding table, which is the first layer of the network. A large vocabulary (like GPT-2’s 50,000 tokens) would create an embedding table with millions of parameters, dwarfing the size of the small model being trained.

2. The Transformer Architecture: The Brains of the Operation

The workshop focuses on a GPT-2-style, decoder-only transformer architecture. While modern architectures have numerous optimizations for scale and context length, the core components remain remarkably consistent.

  • Multi-Head Self-Attention: This is the core mechanism that allows the model to weigh the importance of different tokens in the input sequence when producing an output. It helps the model understand relationships, like how “sky” and “blue” are correlated. Different “heads” can learn to focus on different aspects of the text, such as grammar, punctuation, or semantic concepts.
  • Feed-Forward Network (MLP): After the attention mechanism identifies relationships between tokens, the MLP (Multi-Layer Perceptron) processes this information, organizing the context in a way that helps the model generate the final logits for the next token.
  • Residual Connections: These are simple but crucial additions (x = x + attention(x)). Instead of each layer completely transforming the input, it only adds a modification. This stabilizes training by preventing the model’s internal representations from changing too drastically from one layer to the next.
  • Layer Normalization: This component keeps the activations (the outputs of each layer) within a controlled range. It prevents values from “exploding” as they pass through successive layers, which is another key factor in maintaining training stability.

These components are assembled into a Block, which represents one layer of the transformer. The final model is a stack of these blocks, preceded by an embedding layer and followed by a final linear layer (the LM Head) that produces the probability distribution over the entire vocabulary for the next token.

3. The Training Loop: The Learning Process

The training loop is where the model learns. The objective is next-token prediction, trained using a cross-entropy loss function. This function measures the difference between the model’s predicted probability distribution for the next token and the actual next token in the sequence.

  • Data Loading: The data is split into training and validation sets. The loader creates batches of sequences (e.g., 64 sequences of 256 tokens each) to feed into the model.
  • Optimizer and Learning Rate: The AdamW optimizer is used to update the model’s weights. A critical part of this is the learning rate schedule. Instead of a fixed rate, a warm-up period with a low learning rate is used, followed by a gradual increase to a peak, and then a cosine decay schedule that reduces the rate over the course of training. This allows the model to make large adjustments early on and fine-tune its weights as it gets closer to converging.
  • Monitoring Overfitting: It’s easy for a model to simply memorize the training data, a phenomenon known as overfitting. To detect this, a validation loss (val_loss) is calculated on a set of data the model has never seen. If the training loss continues to decrease while the validation loss starts to increase, the model is overfitting, and its ability to generalize is degrading. This is the signal to stop training.

4. Inference: Generating Text

Once the model is trained, it can generate new text. The process involves feeding the model a prompt and then iteratively sampling the next token from the output probability distribution.

  • Greedy Decoding: The simplest method is to always pick the token with the highest probability. This often leads to repetitive and “boring” text and is rarely used for creative generation.
  • Temperature Sampling: To introduce creativity, a temperature parameter is used to scale the logits before sampling. A higher temperature flattens the probability distribution, making less likely tokens more probable and leading to more random, creative outputs. A temperature of 0.7 is often a good starting point.
  • Top-k Sampling: This method further refines sampling by restricting the choice of the next token to the k most likely candidates, preventing the model from picking highly improbable or nonsensical tokens.
  • Reproducibility: Since sampling is a random process, setting a seed for the random number generator ensures that the same output can be generated every time for a given prompt and checkpoint, which is crucial for debugging and sharing results.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Memory in LLMs: Weights and Activations - Jack Morris, Cornell

Memory in LLMs: Weights and Activations - Jack Morris, Cornell

This talk explores the limitations of current methods for providing knowledge to LLMs, such as large context windows and Retrieval-Augmented Generation (RAG). The speaker argues that the future lies in training knowledge directly into the model's weights. This is achieved through a combination of generating large synthetic datasets from small amounts of source material and using parameter-efficient fine-tuning (PEFT) techniques like LoRA to avoid catastrophic forgetting. The goal is to create more capable, personalized, and efficient models by fundamentally altering how they store and access information.

How We Built a Leading Reasoning Model (Olmo 3)

How We Built a Leading Reasoning Model (Olmo 3)

A comprehensive overview of the entire process behind building Olmo 3 Think, covering the full stack from pre-training architecture and data selection to the detailed post-training recipe involving SFT, DPO, and a deep dive into the advanced infrastructure for scaling Reinforcement Learning (RL). The summary also includes critical reflections on the challenges and nuances of evaluating modern reasoning models.