MskProdComputing
Back to the topic

AI · 10.1

Attention: Learning Where to Look

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 10 marks. Total 20 marks.

Name

Class

Date

Mark  / 20

Section A

Circle the letter of the correct answer. One mark each.

  1. 1In attention, what does a token's query represent?1
    1. AThe position of the token in the sentence
    2. BWhat the token is looking for in other tokens
    3. CThe token's ID number in the vocabulary
    4. DThe information the token passes on
  2. 2How is the score between a query and a key calculated?1
    1. ATheir difference
    2. BTheir dot product
    3. CThe larger of the two
    4. DTheir average
  3. 3Why are the scores divided by √dk?1
    1. ATo make every score a whole number
    2. BTo let the model run on more GPUs
    3. CTo remove the need for a softmax
    4. DTo stop large dot products pushing softmax to extremes
  4. 4After the softmax, what is true of one token's attention weights?1
    1. AEach is between 0 and 1, and they add up to 1
    2. BThey can be any positive or negative number
    3. CEach is a whole number between 1 and 10
    4. DThey add up to the number of tokens
  5. 5Three scaled scores are 2, 0 and 0. Roughly what weight does softmax give the first token?1
    1. A0.33
    2. B0.67
    3. C0.79
    4. D1.00
  6. 6What is the output of attention for one token?1
    1. AThe token's original embedding
    2. BThe key with the highest score
    3. CThe average of all the queries
    4. DA weighted sum of all the value vectors
  7. 7What does a causal mask do?1
    1. AStops each token attending to tokens that come after it
    2. BRemoves rare words from the vocabulary
    3. CBlocks the first token from attending to anything
    4. DHides the labels from the model in testing
  8. 8In cross-attention in a translation model, where do the keys and values come from?1
    1. ARandom noise added for robustness
    2. BThe words the decoder has written so far
    3. CThe encoded source sentence
    4. DA separate dictionary of translations
  9. 9Why does multi-head attention use several heads?1
    1. AOne head is needed for each word in the vocabulary
    2. BEach head handles a different sentence
    3. CExtra heads let the model skip the softmax
    4. DEach head can learn a different kind of relationship
  10. 10A transformer has dmodel = 512 and 8 heads. How many numbers does each head's query have?1
    1. A8
    2. B64
    3. C512
    4. D4,096

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain how scaled dot-product attention calculates the new vector for one token.4
  2. 2ExplainExplain why a transformer that generates text needs a causal mask during training.3
  3. 3DescribeDescribe the roles of the query, key and value vectors in attention.3

Mark scheme

Section A

1B2B3D4A5C6D7A8C9D10B

Section B

1 Explain how scaled dot-product attention calculates the new vector for one token. 4

One mark for each of the following, to a maximum of 4.

  • The token's query is compared with every key by a dot product, giving a score
  • The scores are divided by the square root of d_k
  • Softmax turns the scores into weights between 0 and 1 that add up to 1
  • The output is the weighted sum of the value vectors

Model answer. First the token's query vector is compared with the key vector of every token by taking dot products, which gives a score for each: the more the vectors point the same way, the higher the score. The scores are divided by the square root of the key length, d_k, so they do not grow too large. Softmax then turns the scores into weights between 0 and 1 that add up to 1. Finally each token's value vector is multiplied by its weight and the results are added, so the new vector is a weighted blend of the values, dominated by the most relevant tokens.

2 Explain why a transformer that generates text needs a causal mask during training. 3

One mark for each of the following, to a maximum of 3.

  • In training, the whole sequence is processed at once / in parallel
  • Without a mask a token could attend to later tokens, including the one it must predict, and copy it
  • The mask sets later scores to minus infinity so their weights are 0, matching generation where later tokens do not exist yet

Model answer. During training, a whole sentence is fed in at once so that every position can be trained in parallel. Without a mask, the token at each position could attend to the tokens after it, including the very token it is supposed to predict, so it would learn to copy rather than to predict. The causal mask sets every score for a later position to minus infinity before the softmax, which makes those weights exactly 0, so each token sees only earlier tokens, just as it will when generating, when later tokens do not exist yet.

3 Describe the roles of the query, key and value vectors in attention. 3

One mark for each of the following, to a maximum of 3.

  • The query represents what a token is looking for
  • Keys are matched against the query to decide how relevant each token is / how much attention it gets
  • Values carry the information that is passed on / blended into the output
  • All three are made by multiplying the token's embedding by learned weight matrices

Model answer. Each token's embedding is multiplied by three learned weight matrices to make a query, a key and a value. The query represents what the token is looking for. Every token's key is compared with the query, and how well they match decides how much attention that token receives. The values carry the actual information: the output is a blend of the values, weighted by those attention scores.