Gradient Descent: How Models Learn by Rolling Downhill

Section A

Circle the letter of the correct answer. One mark each.

1. 1What is the gradient descent update for a parameter?1
   1. Aparameter ← learning rate − gradient
   2. Bparameter ← parameter − learning rate × gradient
   3. Cparameter ← parameter + learning rate × gradient
   4. Dparameter ← gradient ÷ learning rate
2. 2What happens if the learning rate is far too large?1
   1. AThe model stops using its training data altogether
   2. BTraining becomes slow but always reaches the minimum
   3. CThe gradients all become exactly zero at once
   4. DSteps overshoot the minimum and the loss bounces or explodes
3. 3A dataset has 60,000 examples and the batch size is 200. How many steps make one epoch?1
   1. A200
   2. B300
   3. C60,000
   4. D12,000,000
4. 4Which is a hyperparameter?1
   1. AThe gradient of the loss
   2. BA bias inside the model
   3. CThe learning rate
   4. DA weight inside the model
5. 5Training loss keeps falling but validation loss has started rising. What is happening?1
   1. AThe data has been scaled twice
   2. BThe learning rate is too small
   3. CThe model is underfitting
   4. DThe model is starting to overfit
6. 6What is a saddle point?1
   1. AA flat point that is low in some directions and high in others
   2. BThe place on the landscape where training always starts
   3. CThe lowest point anywhere in the whole loss landscape
   4. DA point where the loss becomes infinitely large
7. 7Why does mini-batch gradient descent use random batches rather than the whole dataset?1
   1. AThe whole dataset cannot be stored on any computer
   2. BRandom batches guarantee the global minimum is found
   3. CEach step is far cheaper, and the noise can help training
   4. DUsing every example would give the wrong gradient
8. 8How does momentum help in a long narrow ravine?1
   1. AZigzags across the ravine cancel while progress along it builds up
   2. BIt makes the learning rate larger every step
   3. CIt recalculates the gradient on the whole dataset
   4. DIt stops the model from overfitting the data
9. 9What does Adam add to momentum?1
   1. AA second, larger learning rate for the bias
   2. BA way to find the exact minimum in one step
   3. CLabels for data that has none
   4. DA separately adapted step size for every parameter
10. 10Two balls start at different points on the same loss landscape. Why might they end in different places?1
    1. AOnly one can use the training data
    2. BOne of them has a different loss function
    3. CThey can roll into different valleys
    4. DGradient descent is completely random