MskProdComputing
Back to the topic

AI · 5.1

Loss Functions and Gradients: Measuring Wrongness

Section A: 10 multiple choice questions, 10 marks. Section B: 3 written questions, 11 marks. Total 21 marks.

Name

Class

Date

Mark  / 21

Section A

Circle the letter of the correct answer. One mark each.

  1. 1During training, what is the loss a function of?1
    1. AThe time the model has been training
    2. BThe model's parameters, with the data held fixed
    3. CThe number of layers in the model
    4. DThe input data, with the parameters held fixed
  2. 2Which loss is the usual choice for a classifier that outputs probabilities?1
    1. ACross-entropy
    2. BMean absolute error
    3. CAccuracy
    4. DMean squared error
  3. 3A model gives the correct class a probability of 0.5. What is its cross-entropy loss, −ln 0.5?1
    1. AAbout 2.30
    2. BExactly 0.5
    3. CAbout 0.69
    4. DAbout 0.05
  4. 4What happens to cross-entropy loss as the probability given to the right answer approaches 0?1
    1. AIt falls towards 0
    2. BIt grows towards infinity
    3. CIt stays at a constant value
    4. DIt becomes negative
  5. 5What is the derivative of L(w) = (w − 3)² at w = 5?1
    1. A4
    2. B2
    3. C0
    4. D−4
  6. 6The derivative of the loss at the current w is negative. What should training do to w?1
    1. ADecrease it
    2. BSet it to zero
    3. CLeave it unchanged
    4. DIncrease it
  7. 7In which direction does the gradient point?1
    1. AAlong a contour of equal loss
    2. BThe direction of steepest decrease in loss
    3. CThe direction of steepest increase in loss
    4. DTowards the origin of the parameter space
  8. 8What is a partial derivative?1
    1. AThe derivative of the final layer only
    2. BHalf of the full derivative of the loss
    3. CA derivative calculated on part of the data
    4. DThe slope with respect to one parameter, with the others held fixed
  9. 9Why are numerical gradients not used to train large models?1
    1. AThey cannot be calculated on a computer
    2. BThey always give the wrong answer for neural networks
    3. CThey need two loss calculations for every single parameter
    4. DThey only work for models with one parameter
  10. 10What does a contour on a loss contour map join?1
    1. APoints with the same gradient
    2. BModels with the same number of parameters
    3. CThe steps of the training run
    4. DPoints with the same loss

Section B

Answer in the space provided. The number of marks tells you how many separate points to make.

  1. 1ExplainExplain why cross-entropy is a better loss than accuracy for training a classifier.4
  2. 2CalculateFor L(w) = (w − 4)², calculate dL/dw at w = 1 and explain what the answer tells you.3
  3. 3DescribeDescribe what the gradient of a loss function is and how it is used in training.4

Mark scheme

Section A

1B2A3C4B5A6D7C8D9C10D

Section B

1 Explain why cross-entropy is a better loss than accuracy for training a classifier. 4

One mark for each of the following, to a maximum of 4.

  • Cross-entropy uses the predicted probability of the correct class
  • It gives larger loss for confident wrong answers
  • It changes smoothly, so gradients exist / small improvements are rewarded
  • Accuracy only counts right or wrong / is flat so gives no gradient

Model answer. Cross-entropy uses the probability the model gave to the correct class, so it measures how confident the model was as well as whether it was right, and it punishes confident wrong answers very heavily. Because it changes smoothly as the probabilities change, it has a gradient, so even a small improvement lowers the loss and gradient descent knows which way to move. Accuracy only counts right or wrong, so it stays flat while the model improves slightly, giving no gradient to learn from.

2 For L(w) = (w − 4)², calculate dL/dw at w = 1 and explain what the answer tells you. 3

One mark for each of the following, to a maximum of 3.

  • dL/dw = 2(w − 4)
  • At w = 1 it is −6
  • Negative, so the loss falls as w increases: increase w

Model answer. dL/dw = 2(w − 4), so at w = 1 it is 2 × (1 − 4) = −6. The slope is negative, which means the loss decreases as w increases, so to reduce the loss w should be increased, towards the minimum at w = 4. The size, 6, shows the slope is fairly steep, so w is a long way from the minimum.

3 Describe what the gradient of a loss function is and how it is used in training. 4

One mark for each of the following, to a maximum of 4.

  • The vector of partial derivatives of the loss for every parameter
  • It points in the direction of steepest increase in loss
  • Training moves the parameters in the opposite direction
  • By a small step, scaled by the learning rate, repeatedly

Model answer. The gradient is the vector of partial derivatives of the loss with respect to every parameter. It points in the direction in which the loss increases most steeply. In training, each parameter is moved a small step in the opposite direction, along the negative gradient, with the size of the step set by the learning rate. Repeating this many times walks the parameters downhill towards a minimum of the loss.