MskProdComputing
Back to the topic

AI · 11.2

Using and Adapting Large Language Models

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 13 marks. Total 23 marks.

Name

Class

Date

Mark  / 23

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What does greedy decoding do?1
    1. AAlways picks the most likely next token
    2. BPicks the longest available token
    3. CPicks the token used least so far
    4. DPicks a token completely at random
  2. 2What happens to the next-token probabilities when the temperature is lowered below 1?1
    1. AEvery token becomes equally likely
    2. BThe probabilities stop adding up to 1
    3. CThe most likely tokens become even more likely
    4. DThe least likely tokens become more likely
  3. 3With top-p = 0.9, which tokens can be sampled?1
    1. AThe 9 most likely tokens
    2. BOnly tokens with a probability above 0.9
    3. CAny token except the most likely one
    4. DThe most likely tokens until their total reaches 0.9
  4. 4What is few-shot prompting?1
    1. ALimiting the answer to a few words
    2. BTraining the model for a few steps
    3. CIncluding a few worked examples of the task in the prompt
    4. DAsking the same question a few times
  5. 5Why does retrieval-augmented generation reduce made-up answers?1
    1. AIt retrains the model every time a question is asked
    2. BThe model answers from relevant documents placed in its prompt
    3. CIt removes rare words from the vocabulary
    4. DIt sets the model's temperature to zero
  6. 6In RAG, how are the relevant chunks usually found?1
    1. ABy choosing chunks at random
    2. BBy reading every document from start to finish
    3. CBy asking the user to pick them
    4. DBy comparing embeddings of the question and the chunks
  7. 7What does LoRA train instead of the full weight matrices?1
    1. AA brand-new model trained from scratch
    2. BA small low-rank update added to frozen weights
    3. COnly the embedding matrix and biases
    4. DA larger version of the tokenizer
  8. 8A 7 billion parameter model is stored with 4-bit weights. Roughly how much memory do the weights need?1
    1. A28 GB
    2. B7 GB
    3. C14 GB
    4. D3.5 GB
  9. 9What is benchmark contamination?1
    1. AErrors in the benchmark's answer key
    2. BRunning a benchmark on the wrong GPU
    3. CTest questions appearing in the model's training data
    4. DUsing too few questions in a test
  10. 10When is fine-tuning a better choice than prompting or RAG?1
    1. AWhen the model must learn a consistent style or skill from many examples
    2. BWhen you need the model to cite its sources
    3. CWhen the answers depend on documents that change every day
    4. DWhen you have only one example of the task

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain how temperature and top-p sampling change the text a language model produces.4
  2. 2ExplainA school wants a chatbot that answers questions about its own policies. Explain how retrieval-augmented generation would work for this, and give one advantage over fine-tuning.5
  3. 3CompareCompare quantisation and distillation as ways to make a language model cheaper to run.4

Mark scheme

Section A

1A2C3D4C5B6D7B8D9C10A

Section B

1 Explain how temperature and top-p sampling change the text a language model produces. 4

One mark for each of the following, to a maximum of 4.

  • Temperature divides the logits before the softmax
  • A low temperature sharpens the distribution: predictable, repetitive text
  • A high temperature flattens it: more varied, surprising text, with more mistakes
  • Top-p keeps only the most likely tokens until their total probability reaches p
  • This cuts off the unlikely tail, and the kept probabilities are rescaled

Model answer. Temperature divides every logit before the softmax. Below 1 it sharpens the distribution, so the most likely tokens become even more likely and the text becomes predictable and can become repetitive; above 1 it flattens the distribution, so less likely tokens are chosen more often and the text becomes more varied but more error-prone. Top-p sampling keeps only the most likely tokens until their probabilities add up to p, rescales them to sum to 1 and samples from those, so the very unlikely tail of nonsense tokens can never be chosen, while the number of options still adapts to how sure the model is.

2 A school wants a chatbot that answers questions about its own policies. Explain how retrieval-augmented generation would work for this, and give one advantage over fine-tuning. 5

One mark for each of the following, to a maximum of 5.

  • The policy documents are split into chunks
  • Each chunk is turned into an embedding and stored in a vector database
  • The question is embedded and the most similar chunks are retrieved
  • The chunks are added to the prompt with an instruction to answer from them
  • Advantage: when a policy changes, only the documents need updating, with no retraining
  • Advantage: the chatbot can cite which policy its answer came from

Model answer. The school's policy documents are split into chunks of a few hundred words, each chunk is turned into an embedding, and the embeddings are stored in a vector database. When a student asks a question, the question is embedded too, and the chunks with the most similar embeddings are retrieved. Those chunks are placed in the prompt with an instruction to answer only from them and to name the policy used. One advantage over fine-tuning is that when a policy changes, the school only has to update the documents, with no retraining, and the answers can cite the exact policy they came from.

3 Compare quantisation and distillation as ways to make a language model cheaper to run. 4

One mark for each of the following, to a maximum of 4.

  • Quantisation stores the same model's weights with fewer bits
  • It cuts memory and speeds the model up, usually with a small loss of quality
  • Distillation trains a smaller student model to imitate a larger teacher
  • Distillation needs a training run, but gives a model with fewer parameters

Model answer. Quantisation keeps the same model but stores each weight with fewer bits, for example 4-bit integers instead of 16-bit floats, which cuts the memory needed by a factor of four and makes the model faster, usually with only a small loss of quality, and it needs no new training. Distillation instead trains a new, smaller student model to imitate the output probabilities of a large teacher, so the result has far fewer parameters and is cheaper to run, but it requires a full training run and good data, and the student is usually somewhat less capable than its teacher.