MskProdComputing
Back to the topic

AI · 11.1

How Large Language Models Are Built

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 11 marks. Total 21 marks.

Name

Class

Date

Mark  / 21

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What is a base model trained to do during pretraining?1
    1. ASearch the internet for answers
    2. BPredict the next token in a huge amount of text
    3. CFollow instructions from its users
    4. DAnswer questions politely
  2. 2Why is pretraining called self-supervised?1
    1. ANo loss function is used in training
    2. BThe model writes its own training text
    3. CThe labels are the next tokens of the text itself
    4. DEvery example is labelled by a human expert
  3. 3Roughly how many operations does it take to train a transformer with N parameters on D tokens?1
    1. AN × D ÷ 6
    2. BN + D
    3. CN2 × D
    4. D6 × N × D
  4. 4What did the Chinchilla paper (2022) find?1
    1. AMany models had too few training tokens for their size
    2. BTraining data no longer affects the loss
    3. CLarger models always need less data
    4. DModels stop improving beyond a billion parameters
  5. 5What does a scaling law describe?1
    1. AHow quickly a GPU can multiply two matrices
    2. BHow loss falls predictably as compute, data and parameters grow
    3. CHow to split a model across many computers
    4. DHow many tokens fit in a context window
  6. 6What is in-context learning?1
    1. ARetraining the model on each new conversation
    2. BDoing a new task from examples in the prompt, without changing the weights
    3. CLearning only from the context of the training data
    4. DCopying a document into the model's weights
  7. 7What does supervised fine-tuning teach a pretrained model?1
    1. ATo respond in the style of a helpful assistant
    2. BTo use less memory when it runs
    3. CTo read much longer documents
    4. DTo predict the next token for the first time
  8. 8In RLHF, what does the reward model learn?1
    1. ATo generate the next token of an answer
    2. BTo count the tokens in a prompt
    3. CTo translate between two languages
    4. DTo predict which answers people prefer
  9. 9Why is a penalty for drifting away from the instruction-tuned model used in RLHF?1
    1. ATo make training use fewer GPUs
    2. BTo increase the size of the vocabulary
    3. CTo remove the need for human feedback
    4. DTo stop the model gaming the reward model with odd text
  10. 10Why do language models sometimes state false things confidently?1
    1. AThey copy the last website they visited
    2. BThey choose every word completely at random
    3. CThey are trained to produce likely text, which is not always true
    4. DTheir training data contains no facts

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1DescribeDescribe the three stages used to turn a transformer into a chat assistant.5
  2. 2ExplainA company has a fixed compute budget for training a language model. Explain how scaling laws help it choose the model's size and the amount of training data.3
  3. 3CalculateA model has 10 billion parameters and is trained on 200 billion tokens, using 500 GPUs that each sustain 1014 operations per second. Calculate the training time in days.3

Mark scheme

Section A

1B2C3D4A5B6B7A8D9D10C

Section B

1 Describe the three stages used to turn a transformer into a chat assistant. 5

One mark for each of the following, to a maximum of 5.

  • Pretraining: next-token prediction on a vast amount of text
  • This produces a base model that continues text but does not follow instructions
  • Supervised fine-tuning / instruction tuning on example conversations or prompt and response pairs
  • Preference tuning: people compare or rank answers
  • A reward model learns the preferences and reinforcement learning tunes the model to score highly
  • The aim of the later stages is helpful, honest, harmless answers

Model answer. First, pretraining trains the transformer to predict the next token on trillions of tokens of text, which gives a base model that knows a great deal but simply continues text rather than following instructions. Second, supervised fine-tuning, or instruction tuning, trains it further on example conversations made of prompts and ideal responses, so it learns to behave as an assistant. Third, preference tuning such as RLHF has people compare pairs of the model's answers; a reward model learns to predict which answers people prefer, and reinforcement learning then tunes the model towards answers that score highly, making it more helpful and less harmful.

2 A company has a fixed compute budget for training a language model. Explain how scaling laws help it choose the model's size and the amount of training data. 3

One mark for each of the following, to a maximum of 3.

  • Loss falls predictably as a power law with compute, so small runs can predict large ones
  • For a fixed budget, model size and data should be balanced, about 20 tokens per parameter
  • A model that is too large for its data is undertrained / wastes compute
  • A smaller model trained on more data is cheaper to run afterwards

Model answer. Scaling laws show that loss falls smoothly, as a power law, as compute grows, so the company can run a few small, cheap experiments and predict the loss of the full-size model before spending the budget. The Chinchilla result shows that, for a fixed budget, parameters and training tokens should grow together, at roughly 20 tokens per parameter, because a model that is too big for its data is undertrained and wastes compute. The company might even choose a smaller model trained on more tokens than that, since a smaller model is cheaper every time it is used.

3 A model has 10 billion parameters and is trained on 200 billion tokens, using 500 GPUs that each sustain 1014 operations per second. Calculate the training time in days. 3

One mark for each of the following, to a maximum of 3.

  • Compute = 6 × 1010 × 2 × 1011 = 1.2 × 1022 operations
  • Combined speed = 500 × 1014 = 5 × 1016 operations per second
  • Time = 2.4 × 105 seconds ≈ 2.8 days

Model answer. Training compute is about 6 × N × D = 6 × 10,000,000,000 × 200,000,000,000 = 1.2 × 1022 operations. Together the GPUs perform 500 × 1014 = 5 × 1016 operations per second. The time is 1.2 × 1022 ÷ (5 × 1016) = 240,000 seconds, and 240,000 ÷ 86,400 ≈ 2.8 days.