Loss Functions and Gradients: Measuring Wrongness

What a loss function is

To improve, a model needs to know how wrong it is, as one number. That number is the loss (also called the cost or error), and the formula that produces it is the loss function. Training means changing the parameters to make the loss as small as possible.

The crucial shift in thinking: during training the data is fixed and the parameters vary. So the loss is a function of the parameters. Each choice of w gives a different line, and each line has a different loss.

For the revision data from Level 4, with b held at 33.2, a slope of 3 badly underestimates, a slope of 10 badly overestimates, and the loss is lowest at w ≈ 6.5. Plotted against w, the mean squared error makes a bowl, and training means finding the bottom of the bowl.

Common loss functions

Here p is the probability the model gave to "yes", and y is 1 for yes and 0 for no. The loss over a dataset is the average over all the examples.

Why cross-entropy?

Cross-entropy only looks at the probability the model gave to the right answer, and takes the negative logarithm.

Being confidently wrong is punished brutally: the loss heads to infinity as p heads to 0. That pushes a classifier to be honest about uncertainty. A language model's training loss is exactly this, averaged over every next-word prediction it makes.

The loss landscape

With two parameters, w and b, the loss is a surface over the w-b plane: a landscape whose height at each point is the loss of that line.

• Every point on the floor is one possible model.
• The height above it is how wrong that model is.
• Training is a search for the lowest point.

For linear regression the landscape is a single smooth bowl. For a neural network it is a landscape in millions or billions of dimensions, with valleys, ridges, plateaus and saddles. Nobody can picture that, but the maths of finding the way down is the same in any number of dimensions.

A contour map of the landscape, like a walking map of a hill, joins points of equal loss. The rings close in on the minimum.

Slopes and derivatives

To walk downhill you need to know which way is down. In one dimension, that is the slope, and in calculus the slope of a curve at a point is its derivative.

For the loss L(w) = (w − 3)², the derivative is dL/dw = 2(w − 3):

So the derivative tells us two things: its sign says which way is uphill, so we move the other way, and its size says how steep the slope is.

Three rules you will use

• Power rule: the derivative of x² is 2x, of x³ is 3x², of xⁿ is nx to the power n − 1.
• Constant multiple: the derivative of 5x² is 5 × 2x = 10x.
• Chain rule: for a function inside a function, multiply the derivatives. The derivative of (w − 3)² is 2(w − 3) × 1. Level 7 shows that backpropagation is the chain rule applied over and over.

Gradients: slopes in many directions

With many parameters, we take the slope in each direction separately. Each one is a partial derivative, written ∂L/∂w, "the rate the loss changes as w changes, with everything else held still". Collected into a vector, they make the gradient, written ∇L.

The gradient of the mean squared error

For the line prediction = wx + b, with error = prediction − y for each example:

These are easy to read. If the predictions are too high, the errors are positive, both gradients are positive, and stepping against them lowers w and b. The bigger an example's x, the more it pushes on w.

Numerical gradients: checking with tiny nudges

You can estimate a derivative without any calculus: nudge the parameter a tiny amount h up and down, and see how much the loss changes.

This is how engineers check that their gradient code is correct, called gradient checking. But it is far too slow to train with: a model with a billion parameters would need two billion loss calculations for a single step. Backpropagation (Level 7) gets every partial derivative at once for the cost of about one extra pass through the model, which is what makes deep learning possible at all.

Hands-on: losses and gradients in code

The loss curve, drawn in text

hours = [1, 2, 2, 3, 4, 5, 6, 6, 7, 8]
score = [38, 45, 50, 52, 61, 64, 70, 75, 78, 86]
b = 33.2

def loss(w):
    return sum((w * x + b - y) ** 2 for x, y in zip(hours, score)) / len(hours)

for w in [3, 4, 5, 6, 6.5, 7, 8, 9, 10]:
    L = loss(w)
    print(f"w = {w:4}: loss {L:7.1f} " + "#" * int(L / 10))

Calculus versus nudging

def L(w):
    return (w - 3) ** 2

def exact_slope(w):
    return 2 * (w - 3)

def nudge_slope(w, h=0.0001):
    return (L(w + h) - L(w - h)) / (2 * h)

for w in [1, 3, 5, 10]:
    print(f"w = {w:2}: exact {exact_slope(w):6.2f}   by nudging {nudge_slope(w):6.2f}")

The gradient of the revision model

hours = [1, 2, 2, 3, 4, 5, 6, 6, 7, 8]
score = [38, 45, 50, 52, 61, 64, 70, 75, 78, 86]

def gradients(w, b):
    n = len(hours)
    dw = sum(2 * (w * x + b - y) * x for x, y in zip(hours, score)) / n
    db = sum(2 * (w * x + b - y) for x, y in zip(hours, score)) / n
    return dw, db

for w, b in [(0, 0), (10, 40), (6.52, 33.23)]:
    dw, db = gradients(w, b)
    print(f"at w={w:5}, b={b:5}: dL/dw = {dw:8.2f}, dL/db = {db:7.2f}")

At (0, 0) both gradients are large and negative, so increasing w and b reduces the loss. At (10, 40) they are positive: the line is too high. At the least squares answer they are almost exactly zero, which is how you recognise the bottom of the bowl.

Cross-entropy

import math

for p in [0.99, 0.9, 0.7, 0.5, 0.3, 0.1, 0.01]:
    print(f"p(correct) = {p:4}: loss = {-math.log(p):.3f}")

Level check

• Explain what a loss function is and why it is a function of the parameters
• Choose a suitable loss for regression and for classification
• Calculate cross-entropy for a prediction and explain why confident mistakes cost so much
• Describe a loss landscape and what its lowest point represents
• Calculate a derivative with the power rule and explain what its sign and size mean
• Explain what the gradient is and why we step in the opposite direction

Key terms you must be able to define

Loss function

A formula giving one number that measures how wrong a model's predictions are; training minimises it.

Mean squared error

The average of the squared differences between predictions and true values.

Cross-entropy

A classification loss: the negative logarithm of the probability given to the correct answer.

Loss landscape

The surface formed by the loss over every possible setting of the parameters.

Minimum

The lowest point of the loss: the best parameters for the training data.

Derivative

The slope of a function at a point: how fast its output changes as its input changes.

Partial derivative

The slope with respect to one parameter while all the others are held fixed.

Gradient

The vector of all the partial derivatives; it points in the direction of steepest increase.

Chain rule

The derivative of a function of a function is the product of their derivatives.

Gradient checking

Comparing gradients from code with estimates made by nudging each parameter slightly.

Flashcards

Go further: notes for future AI engineers

Where cross-entropy comes from

Cross-entropy comes from information theory, founded by Claude Shannon in 1948. The information in an event with probability p is −log₂ p bits: a fair coin flip carries 1 bit, a 1 in 1,024 event carries 10. Cross-entropy measures how many bits, on average, you would need to encode what really happened using the model's probabilities. A perfect model needs the fewest. So when a language model's loss falls, it is literally becoming a better compressor of text, which is why some researchers describe prediction and compression as two sides of the same coin.

Maximum likelihood

Minimising cross-entropy is the same as choosing the parameters that make the training data most probable under the model, called maximum likelihood estimation. Minimising mean squared error is maximum likelihood too, if the noise is assumed to follow a normal distribution. Behind the formulas sits one idea: choose the model under which what you saw was most likely.

Misconceptions to avoid