AI · 10.2

Transformers: The Architecture Behind Modern AI

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 11 marks. Total 21 marks.

Name

Class

Date

Mark / 21

Section A

Circle the letter of the correct answer. One mark each.

1What did the 2017 transformer remove that earlier translation models relied on?1
1. AWord embeddings
2. BRecurrence: processing tokens one at a time
3. CTraining with gradient descent
4. DThe softmax function
2Why does a transformer need positional encoding?1
1. ATo decide which tokens to mask out
2. BTo make the training data more varied
3. CAttention on its own ignores word order
4. DTo reduce the size of the vocabulary
3What does the feed-forward network in a transformer block do?1
1. AProcesses each token's vector on its own
2. BMoves information between tokens
3. CChooses which tokens to mask out
4. DConverts tokens into ID numbers
4What does a residual connection do in a transformer block?1
1. AAdds the sublayer's input to its output
2. BRemoves tokens that are not needed
3. CSends the output back to the first layer
4. DMultiplies the output by the learning rate
5Why must a block's output have the same shape as its input?1
1. ASo that softmax can be skipped
2. BSo that the model uses less memory
3. CSo that the vocabulary stays the same size
4. DSo that blocks can be stacked one on top of another
6Which family of transformer does BERT belong to?1
1. ARecurrent
2. BEncoder-only
3. CDecoder-only
4. DEncoder-decoder
7Which kind of transformer is trained to predict the next token using masked attention?1
1. AEncoder-only, such as BERT
2. BA convolutional network, such as ResNet
3. CAn LSTM, such as early Google Translate
4. DDecoder-only, such as GPT
8How does a Vision Transformer read an image?1
1. AIt looks only at the colour of the centre pixel
2. BIt reads the pixels one at a time from left to right
3. CIt turns the image into a written description first
4. DIt cuts the image into patches and treats each patch as a token
9A context grows from 1,000 to 4,000 tokens. Roughly how many times bigger does the attention matrix get?1
1. A4 times
2. B16 times
3. C2 times
4. DIt stays the same size
10In GPT-2 small, which part holds the most parameters?1
1. AThe final layer normalisation
2. BThe attention layers in its blocks
3. CThe feed-forward networks in its blocks
4. DThe token embedding table

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

1DescribeDescribe the structure of one transformer block.4
2ExplainExplain why transformers replaced recurrent neural networks for language tasks.4
3CompareCompare encoder-only and decoder-only transformers.3

Mark scheme

Section A

1B2C3A4A5D6B7D8D9B10C

Section B

1 Describe the structure of one transformer block. 4

One mark for each of the following, to a maximum of 4.

Multi-head self-attention, in which tokens exchange information
A residual connection adds each sublayer's input to its output
Layer normalisation rescales each token's vector
A feed-forward network is applied to each token separately
The output has the same shape as the input, so blocks can be stacked

Model answer. A transformer block has two sublayers. The first is multi-head self-attention, in which every token gathers information from the other tokens. The second is a feed-forward network, two linear layers with an activation between them, applied to each token's vector separately. Each sublayer is wrapped in a residual connection that adds its input to its output, and in layer normalisation, which rescales each token's vector to mean 0 and standard deviation 1. The output has the same shape as the input, one vector per token, so many identical blocks can be stacked.

2 Explain why transformers replaced recurrent neural networks for language tasks. 4

One mark for each of the following, to a maximum of 4.

Transformers process every token in parallel, using matrix multiplications that suit GPUs
RNNs must process tokens one after another, which is slow to train
Attention connects any two tokens directly, in one step
RNNs forget distant words because of vanishing gradients
Transformers keep improving as they are scaled up with more data and computing power

Model answer. An RNN must read a sequence one token after another, because each hidden state depends on the one before, so training cannot be spread across a GPU and is slow. A transformer processes every token at once with large matrix multiplications, which GPUs do extremely quickly, so it can be trained on far more data. Attention also connects any two tokens directly in a single step, whereas in an RNN information from a distant word has to pass through many steps and gradients vanish, so it is forgotten. Because transformers train efficiently, they could be scaled up, and their performance kept improving with size.

3 Compare encoder-only and decoder-only transformers. 3

One mark for each of the following, to a maximum of 3.

In an encoder-only model every token attends to every other token / full attention
Encoder-only models such as BERT are trained to fill in masked words and used for understanding tasks such as classification
Decoder-only models use a causal mask, so each token sees only earlier tokens
Decoder-only models such as GPT are trained to predict the next token and used for generation

Model answer. Both are built from the same transformer blocks. In an encoder-only model such as BERT, every token can attend to every other token, before and after it, and the model is trained by filling in words that have been hidden, so it is suited to understanding tasks such as classifying text or searching. A decoder-only model such as GPT uses a causal mask, so each token sees only the tokens before it, and it is trained to predict the next token, which makes it suited to generating text one token at a time.