MskProdComputing
Back to the topic

AI · 9.2

Sequence Models: Predicting the Next Word

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 12 marks. Total 22 marks.

Name

Class

Date

Mark  / 22

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What does a language model predict?1
    1. AThe exact sentence a person will say next
    2. BThe probability of each possible next token
    3. CThe meaning of each word in a dictionary
    4. DThe grammar rules of the language
  2. 2What is autoregressive generation?1
    1. AProducing one token at a time and feeding each one back in
    2. BGenerating the whole text in a single step
    3. CCorrecting the text after it has been written
    4. DTranslating text back into its original language
  3. 3'the' appears 10 times before another word, followed by 'cat' 4 times. What is P(cat | the) in a bigram model?1
    1. A0.04
    2. B0.1
    3. C0.4
    4. D4
  4. 4Why can n-gram models not use long contexts?1
    1. AWords further back have no effect on the next word
    2. BMost long word sequences never appear, so the counts are zero
    3. CLong contexts always make the predictions less accurate
    4. DComputers cannot store sequences longer than three words
  5. 5Model A has perplexity 12 on a test text and model B has 45. Which is better?1
    1. AModel B
    2. BThey are equal
    3. CIt cannot be told
    4. DModel A
  6. 6What does an RNN carry from one step to the next?1
    1. AThe final answer
    2. BA new set of weights for each step
    3. CA hidden state vector that acts as its memory
    4. DA copy of the whole dataset
  7. 7Why do plain RNNs struggle with long sentences?1
    1. AThey have no way to output more than one word
    2. BGradients vanish over many steps, so early words are forgotten
    3. CThey cannot read more than ten words at a time
    4. DThey use different weights for every word in the text
  8. 8What does the forget gate in an LSTM do?1
    1. ADecides which parts of the cell state to erase
    2. BResets the model after each sentence
    3. CDeletes training examples
    4. DChooses the next word
  9. 9What was the bottleneck in early sequence-to-sequence translation?1
    1. AThe vocabulary held fewer than one hundred words
    2. BThe encoder could not read punctuation marks
    3. CThe whole source sentence had to fit into one fixed-size vector
    4. DThe decoder could only produce one word per sentence
  10. 10Who first generated text with word statistics, in 1948?1
    1. AAlan Turing
    2. BJohn McCarthy
    3. CGeoffrey Hinton
    4. DClaude Shannon

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain how a language model generates a paragraph of text.4
  2. 2CompareCompare a bigram model with a recurrent neural network as language models.4
  3. 3ExplainExplain the bottleneck problem in sequence-to-sequence models and how attention addressed it.4

Mark scheme

Section A

1B2A3C4B5D6C7B8A9C10D

Section B

1 Explain how a language model generates a paragraph of text. 4

One mark for each of the following, to a maximum of 4.

  • It predicts a probability for every possible next token
  • It chooses / samples one token
  • The chosen token is added to the input / context
  • Repeated one token at a time until finished: autoregressive

Model answer. Given the text so far, the language model predicts a probability for every token in its vocabulary being the next one. It then chooses one token, usually by sampling from those probabilities. The chosen token is appended to the text, and the longer text becomes the input for the next prediction. Repeating this loop one token at a time, which is called autoregressive generation, builds up the whole paragraph.

2 Compare a bigram model with a recurrent neural network as language models. 4

One mark for each of the following, to a maximum of 4.

  • A bigram model counts pairs of words / uses only the previous word
  • It suffers from sparsity / cannot use long context
  • An RNN carries a hidden state summarising all previous words
  • RNNs generalise through learned weights but forget over long sequences / vanishing gradients

Model answer. A bigram model simply counts how often each word follows each other word, so it predicts the next word from only the previous one, and it cannot handle combinations it never saw because their counts are zero. A recurrent neural network instead reads the sequence one token at a time and carries a hidden state vector that summarises everything so far, so in principle it can use the whole context, and its learned weights let it generalise to new combinations. However, plain RNNs suffer from vanishing gradients, so in practice they forget words more than a few dozen steps back, which LSTMs improved.

3 Explain the bottleneck problem in sequence-to-sequence models and how attention addressed it. 4

One mark for each of the following, to a maximum of 4.

  • The encoder compresses the whole input into one fixed-size vector
  • Information is lost for long sentences
  • Attention lets the decoder look at all encoder states
  • Weighted by how relevant each is at each step

Model answer. In a sequence-to-sequence model the encoder reads the whole source sentence and squeezes it into its final hidden state, a single fixed-size vector, which is all the decoder receives. However long the sentence, its meaning must pass through that one vector, so information is lost and translation quality falls for long sentences. Attention removes the bottleneck by letting the decoder look back at every one of the encoder's hidden states at each step, weighting them by how relevant each is to the word being produced, so it can focus on the right part of the source.