MskProdComputing
Back to the topic

AI · 7.2

Backpropagation: How a Network Learns

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 12 marks. Total 22 marks.

Name

Class

Date

Mark  / 22

Section A

Circle the letter of the correct answer. One mark each.

  1. 1What does backpropagation compute?1
    1. AThe best learning rate for the network
    2. BThe output of the network for a new input
    3. CThe gradient of the loss with respect to every weight
    4. DThe number of layers the network needs
  2. 2If dy/du = 3 and du/dx = 4, what is dy/dx?1
    1. A0.75
    2. B1
    3. C7
    4. D12
  3. 3In which direction does the backward pass travel through the network?1
    1. AFrom the loss back towards the inputs
    2. BFrom the inputs towards the loss
    3. CIn a random order each time
    4. DFrom the middle layer outwards
  4. 4In the graph u = w × x with x = 2, a gradient of 0.2 arrives at u. What is dL/dw?1
    1. A0.1
    2. B0.2
    3. C0.4
    4. D2.2
  5. 5What is the local derivative of a sigmoid node whose output is a?1
    1. Aa(1 − a)
    2. B1
    3. Ca²
    4. D1 − a
  6. 6Roughly how much does a backward pass cost compared with a forward pass?1
    1. ANothing: it is free
    2. BAbout a million times as much
    3. CExactly the same
    4. DAbout twice as much
  7. 7Why do networks start with random weights rather than all zeros?1
    1. ARandom weights mean no training is needed at all
    2. BIdentical weights would make every neuron learn the same thing
    3. CZero weights would make training far too fast to control
    4. DComputers cannot store a weight of exactly zero
  8. 8A training run's loss suddenly jumps to NaN. What is a likely cause?1
    1. AToo few epochs
    2. BThe test set being too small
    3. CVanishing gradients
    4. DExploding gradients
  9. 9What does loss.backward() do in PyTorch?1
    1. AResets every gradient back to zero
    2. BRuns backpropagation to compute every parameter's gradient
    3. CRuns the network's forward pass on new data
    4. DUpdates every weight using gradient descent
  10. 10With a sigmoid output and cross-entropy loss, what is the gradient of the loss with respect to the output's z?1
    1. Ay − p
    2. Bp − y
    3. Cp(1 − p)
    4. D2(p − y)

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain what backpropagation is and why it is needed to train neural networks.4
  2. 2CalculateA neuron computes z = wx + b and a = σ(z), with loss L = (a − y)². Given dL/da = −0.5 and a = 0.75, calculate dL/dw when x = 2.4
  3. 3DescribeDescribe the steps of the training loop used to train a neural network.4

Mark scheme

Section A

1C2D3A4C5A6D7B8D9B10B

Section B

1 Explain what backpropagation is and why it is needed to train neural networks. 4

One mark for each of the following, to a maximum of 4.

  • Training needs the gradient of the loss for every weight
  • Calculating each separately / by nudging would be far too slow
  • Backpropagation applies the chain rule backwards from the loss
  • It gets all the gradients in one backward pass, about twice the cost of a forward pass

Model answer. Gradient descent needs the gradient of the loss with respect to every weight in the network, and a network can have millions of weights. Estimating each gradient separately, by nudging one weight at a time, would need millions of forward passes per step and be far too slow. Backpropagation applies the chain rule backwards through the network, from the loss towards the inputs, multiplying local derivatives at each layer. It produces every gradient in a single backward pass, costing only about twice a forward pass, which makes training large networks practical.

2 A neuron computes z = wx + b and a = σ(z), with loss L = (a − y)². Given dL/da = −0.5 and a = 0.75, calculate dL/dw when x = 2. 4

One mark for each of the following, to a maximum of 4.

  • da/dz = a(1 − a) = 0.75 × 0.25 = 0.1875
  • dL/dz = −0.5 × 0.1875 = −0.09375
  • dz/dw = x = 2
  • dL/dw = −0.1875

Model answer. The sigmoid's local derivative is da/dz = a(1 − a) = 0.75 × 0.25 = 0.1875. By the chain rule, dL/dz = dL/da × da/dz = −0.5 × 0.1875 = −0.09375. Since z = wx + b, dz/dw = x = 2, so dL/dw = −0.09375 × 2 = −0.1875.

3 Describe the steps of the training loop used to train a neural network. 4

One mark for each of the following, to a maximum of 4.

  • Forward pass: compute the outputs for a batch
  • Compute the loss by comparing outputs with targets
  • Backward pass: backpropagation computes the gradients
  • Update the weights with gradient descent / an optimiser, then repeat for batches and epochs

Model answer. For each batch of training examples, the network first does a forward pass, computing its outputs layer by layer. The loss is calculated by comparing those outputs with the targets. A backward pass then uses backpropagation to compute the gradient of the loss with respect to every weight and bias. Finally an optimiser such as gradient descent or Adam updates every parameter a small step against its gradient. The loop repeats for every batch, and the whole dataset is passed through many times, once per epoch.