How Large Language Models Are Built

What makes a model large

A large language model (LLM) is a decoder-only transformer, exactly as in Level 10, trained to predict the next token on an enormous amount of text. "Large" describes three things at once: the number of parameters, the number of training tokens, and the compute needed to train it.

Turning one of these into a helpful assistant takes three stages:

Stage 1: pretraining

Pretraining is next-token prediction on as much good text as can be found: web pages, books, encyclopaedias, scientific papers and program code. Before training, the data is cleaned: duplicates are removed, low-quality pages are filtered out, and attempts are made to strip personal information and harmful content.

Every position in every document is a training example, because the label is simply the next token. No human has to label anything, which is why this is called self-supervised learning. A trillion-token dataset is a trillion examples.

To predict the next token well across that much text, a model has to absorb grammar, facts, styles, and patterns of reasoning. Predicting the end of "The capital of France is" needs a fact; predicting the end of a proof needs some maths. The result is a base model: it continues text brilliantly, but it is not yet an assistant.

Counting the compute

Training a transformer with N parameters on D tokens takes about

For GPT-3: 6 × 175 billion × 300 billion ≈ 3.2 × 10²³ operations. A modern GPU might sustain about 10¹⁴ operations per second in training, so that is around 3.2 × 10⁹ seconds: about 100 years on one GPU, or roughly 36 days on 1,000 GPUs working together.

That is why frontier models are trained on clusters of thousands of GPUs for months, and why the electricity and hardware cost millions of pounds. The model, its training data and its optimiser state are split across the machines in several ways at once, which Level 14 returns to.

Scaling laws

In 2020, researchers at OpenAI published a striking result: the loss of a language model falls smoothly and predictably as you increase the parameters, the data or the compute, following a power law. On a log-log graph, the trend is a straight line, so a few small, cheap training runs predict how well a far bigger one will do.

In 2022, DeepMind's Chinchilla paper showed that most large models had been undertrained: for a fixed compute budget, it is better to train a smaller model on more tokens, at roughly 20 tokens per parameter. Chinchilla, with 70 billion parameters and 1.4 trillion tokens, beat the 280 billion parameter Gopher, which had used about the same compute on only 300 billion tokens.

Many models now go far beyond 20 tokens per parameter on purpose. A smaller model is cheaper every time anyone uses it, so it is worth spending extra training compute to make it as good as possible: Llama 3's 8 billion parameter model saw about 1,900 tokens per parameter.

In-context learning

The GPT-3 paper was titled "Language Models are Few-Shot Learners". It showed that a large enough model can do a new task from a few examples written into the prompt, with no change to its weights:

This in-context learning gets better with scale, and some abilities seem to appear only once models are large enough. Researchers still debate how sudden these jumps really are: part of the suddenness turned out to come from how the abilities were scored.

Stage 2: instruction tuning

Ask a base model "What is the capital of France?" and it may well continue with more quiz questions, because that is what such a line is usually followed by on the web. It predicts text; it does not follow instructions.

Supervised fine-tuning (SFT), also called instruction tuning, continues training on tens of thousands to millions of example conversations: a prompt, then an ideal response, written or checked by people. The loss is the same next-token loss, applied only to the responses.

The model learns the format of being an assistant: answer the question, follow the instruction, stop when finished. Most of what it knows still comes from pretraining.

Stage 3: learning from preferences

Some qualities are easier to recognise than to write down: which of two explanations is clearer, kinder or more honest. Reinforcement learning from human feedback (RLHF) uses exactly that:

1. The model writes several answers to the same prompt.
2. People compare them and pick the better one.
3. A reward model, another network, learns to predict which answers people prefer.
4. The LLM is fine-tuned with reinforcement learning to write answers the reward model scores highly, with a penalty for drifting too far from the instruction-tuned model, so that it cannot game the reward model with strange text.

In 2022, OpenAI reported that people preferred the answers of a 1.3 billion parameter model trained this way, InstructGPT, to those of the original 175 billion parameter GPT-3. Newer methods such as DPO (2023) learn directly from the preference pairs without a separate reward model, and the feedback can also come from AI models judging answers against written principles.

Since 2024, reasoning models have added another kind of reinforcement learning: on maths and programming problems whose answers can be checked automatically, the model is rewarded for reaching the correct answer, and it learns to write out long chains of reasoning before it answers.

Why language models make things up

A language model is trained to produce likely text, not true text. When it does not know something, the most likely-sounding continuation is often a confident, specific answer, because confident answers are common in its training data. This is called hallucination.

• Facts that appeared rarely in training are stored weakly and are easily confused.
• The model knows nothing after its training data was collected, its knowledge cutoff.
• Unless it was trained to say "I don't know", it rarely does.

Defences include training models to admit uncertainty, letting them look things up (Lesson 11.2), asking them to cite sources, and checking anything important yourself.

Hands-on: the numbers behind LLMs

Training compute

models = [("GPT-3", 175e9, 300e9), ("Gopher", 280e9, 300e9),
          ("Chinchilla", 70e9, 1.4e12), ("Llama 3 8B", 8e9, 15e12)]
gpu = 1e14          # operations per second one modern GPU might sustain in training

for name, n, d in models:
    flops = 6 * n * d
    gpu_years = flops / gpu / (365 * 24 * 3600)
    print(f"{name:11} {n / 1e9:5.0f}B parameters, {d / 1e12:5.2f}T tokens: "
          f"{flops:.1e} operations, {gpu_years:5.0f} GPU-years, {d / n:7.1f} tokens per parameter")

Fitting a scaling law

Five small training runs are enough to fit the straight line on log-log axes, and the line predicts runs a thousand times bigger.

import math, random

random.seed(2)
compute = [1e18, 1e19, 1e20, 1e21, 1e22]                       # operations used by five small runs
loss = [60 * c ** -0.07 * random.uniform(0.98, 1.02) for c in compute]

# A power law is a straight line on log-log axes: log(loss) = log(a) - alpha * log(compute)
xs = [math.log10(c) for c in compute]
ys = [math.log10(v) for v in loss]
n = len(xs)
mx, my = sum(xs) / n, sum(ys) / n
slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
intercept = my - slope * mx
print(f"fitted exponent alpha = {-slope:.3f}")
print(f"each 10x more compute multiplies the loss by {10 ** slope:.3f}")
for c in [1e23, 1e24, 1e25]:
    print(f"predicted loss at {c:.0e} operations: {10 ** (intercept + slope * math.log10(c)):.2f}")

A reward model from people's choices

This is the heart of RLHF in miniature. Each answer is described by three features, and people have compared pairs of answers. The model learns weights so that preferred answers score higher, using the same logistic idea as Level 4.

import math

# Each answer described by three features: [correct, polite, length in hundreds of words]
answers = {"A": [1, 1, 0.8], "B": [1, 0, 0.5], "C": [0, 1, 1.2], "D": [0, 0, 0.3], "E": [1, 1, 2.5]}
# People compared two answers at a time and picked a winner: (winner, loser)
preferences = [("A", "B"), ("A", "C"), ("B", "D"), ("C", "D"),
               ("A", "D"), ("E", "C"), ("A", "E"), ("B", "C")]

w = [0.0, 0.0, 0.0]
def reward(x):
    return sum(wi * xi for wi, xi in zip(w, x))

for epoch in range(300):
    for win, lose in preferences:
        xw, xl = answers[win], answers[lose]
        p = 1 / (1 + math.exp(-(reward(xw) - reward(xl))))    # chance the model agrees with the person
        for i in range(3):
            w[i] += 0.1 * ((1 - p) * (xw[i] - xl[i]) - 0.01 * w[i])

print("learned weights: correct %.2f, polite %.2f, length %.2f" % tuple(w))
for name, x in sorted(answers.items(), key=lambda kv: -reward(kv[1])):
    print(f"answer {name}: reward {reward(x):5.2f}")

Nobody told the reward model that correctness matters most, or that rambling is bad. It worked both out from which answers people chose.

Try this

1. Use 6 × N × D to estimate the compute for a 405 billion parameter model trained on 15 trillion tokens. How many GPU-years is that?
2. Add a preference ("E", "B"), saying a long, correct, polite answer beats a short, correct, rude one. How do the learned weights change?

Level check

• Describe pretraining, and explain why it is self-supervised
• Estimate training compute with 6 × N × D, and turn it into GPU time
• Explain what a scaling law shows, and the Chinchilla rule of about 20 tokens per parameter
• Explain in-context learning
• Describe instruction tuning and why a base model needs it
• Describe the four steps of RLHF, including the reward model
• Explain why language models hallucinate, and ways to reduce it

Key terms you must be able to define

Large language model (LLM)

A transformer with billions of parameters trained to predict the next token on vast amounts of text.

Pretraining

The first and largest stage of training: next-token prediction on trillions of tokens.

Base model

A pretrained model that continues text but has not been tuned to follow instructions.

Self-supervised learning

Learning from labels that come from the data itself, such as the next token of a text.

Training compute

The number of arithmetic operations used in training, about 6 × parameters × tokens for a transformer.

Scaling law

A power law showing how loss falls predictably as parameters, data and compute grow.

Compute-optimal

The best balance of model size and data for a compute budget: about 20 tokens per parameter.

In-context learning

Doing a new task from examples in the prompt, without changing the model's weights.

Supervised fine-tuning (SFT)

Further training on example conversations so the model responds as an assistant.

RLHF

Reinforcement learning from human feedback: tuning a model towards answers people prefer.

Reward model

A network trained on people's comparisons to score how good an answer is.

Hallucination

A fluent, confident statement that is false or unsupported.

Flashcards

Go further: notes for future AI engineers

Training once, serving forever

Training happens once, but inference, running the model to answer people, happens every time anyone uses it. For a popular model, the total compute spent serving users can exceed the training compute. That is why companies deliberately "over-train" smaller models far beyond the Chinchilla ratio: every extra point of quality in a small model is paid for once, and saved on every request.

Synthetic data

High-quality human text is running short, so models are increasingly trained on text generated by other models and then filtered, especially for maths and code, where answers can be checked. Done carelessly, training generation after generation on model output loses the rare and unusual parts of the original data. A 2024 paper in Nature called this model collapse.

Test-time compute

Reasoning models showed a second way to scale. Instead of only making the model bigger, let it think longer: spend more tokens of reasoning on harder problems. Accuracy on difficult maths and programming problems keeps rising with the amount of thinking allowed, so a model can trade answer speed for answer quality.

Misconceptions to avoid