Inside GPT – The Maths Behind the Magic • Alan Smith • GOTO 2024

Large Language Models (LLMs) like GPT can appear magical, but their core function is surprisingly straightforward: given a sequence of tokens, they predict the probability of the next token in that sequence. The generation of a full sentence or paragraph is an iterative process where the model predicts one token at a time, appends it to the input sequence, and runs the entire new sequence through the model again.

The End-to-End LLM Pipeline

When a user interacts with an LLM through an API, the prompt goes through several stages before a response is generated:

1. Service Layer: The request first hits a service layer that handles concerns like billing, content filtering, monitoring, and logging.
2. Tokenization: The raw text prompt is converted into a sequence of integers, known as tokens.
3. The Model: The tokens are fed into the core model, which consists of code (the algorithm) and weights (the learned parameters). The model’s output is a set of logits—a raw, unnormalized array of numbers representing the relative probabilities for every possible next token.
4. Sampling: A separate process from the model itself, sampling uses the logits to select the next token. This is where parameters like temperature and top_p are applied to influence the creativity and randomness of the output.
5. Iteration: The process repeats, feeding the new sequence back into the model until a complete response is formed.

Tokenization: The Language of LLMs

Neural networks don’t process text directly; they process numbers. Tokenization is the crucial step of converting text into integers that the model can understand.

• What are Tokens? Tokens are common sequences of characters found in the model’s training data. They can be whole words (e.g., cat), word fragments (e.g., ization), or even single characters. Most word tokens are prefixed with a space to improve efficiency.
• Language Efficiency: The effectiveness of tokenization varies significantly by language. English is highly efficient because it constitutes the majority of the training data. Other languages, like Swedish or Telugu, require more tokens to represent the same text, which can impact performance, cost, and accuracy. This efficiency gap is a strong argument for why “English is the new programming language” in the context of prompting LLMs.
• Glitch Tokens: The tokenization vocabulary can contain strange artifacts from the training data, such as the infamous Solid Gold Magikarp token. This token exists because a prolific Reddit user with that name was highly active on a subreddit included in the training corpus, even though the content of that subreddit was later filtered out from the model’s training.

From Tokens to Vectors: Embeddings and Position

The model converts integer tokens into high-dimensional floating-point vectors called embeddings.

• Token Embeddings: The model contains a massive lookup table where each token ID maps to a vector (e.g., in GPT-2, a vector with 768 dimensions). This vector captures the semantic meaning of the token, so similar concepts have similar vectors.
• Positional Embeddings: Since sequence order is critical, a separate positional vector is added to each token embedding. This encodes the token’s position in the sequence, allowing the model to differentiate between “the film was not bad, in fact it was very good” and “the film was not good, in fact it was very bad.”

The Transformer and The Attention Mechanism

The core of GPT’s architecture is the decoder-only Transformer, introduced in the paper “Attention Is All You Need.” The key innovation is the attention mechanism.

Attention allows the model to dynamically weigh the importance of different words in the input sequence when producing the next token. It calculates how each word relates to every other word, enabling it to handle long-term dependencies. For example, in the sentence “The cat sat on the floor, it had just been cleaned,” the attention mechanism helps the model understand that “it” refers to the “floor,” not the “cat,” by shifting the vector representation of “it” to be closer to “floor” and “cleaned.” This is what gives the model its powerful ability to understand context.

Practical Considerations

• Prompt Engineering: Crafting detailed system prompts that define a persona, rules, and output format is essential for guiding the model’s behavior.
• Retrieval-Augmented Generation (RAG): To answer questions about data not in its training set, a model can use RAG. This involves retrieving relevant information from an external source (like a search engine or internal documents) and inserting it into the prompt, giving the model the context it needs.
• Statelessness and Cost: LLMs are stateless. To maintain a conversation, the entire chat history must be sent with every new turn. This causes the token count to grow, increasing costs and eventually hitting the model’s context limit.

• Tags:
• Large Language Models
• GPT
• Transformer Architecture
• Tokenization
• Embeddings
• RAG
• Attention Mechanism