Gradient Descent: How Models Learn by Rolling Downhill
The algorithm
Imagine standing on a mountain in thick fog, trying to reach the valley floor. You cannot see the valley, but you can feel the slope under your feet. So you take a step in the steepest downhill direction, feel the slope again, and repeat.
Repeat: parameters ← parameters − learning rate × gradient
- Start with random (or zero) parameters.
- Calculate the loss and its gradient on the training data.
- Move every parameter a small step against its gradient.
- Repeat until the loss stops improving.
Step 1 of 9 Start at w = −2, high on the left: the slope is steep
Press Play to watch it work. Far from the minimum the slope is steep, so the steps are big. Close to it the slope flattens, so the steps shrink on their own, and the ball settles gently into the bottom.
The learning rate
The learning rate, often written η (eta), sets the size of each step. It is the most important setting in all of deep learning.
| Learning rate | What happens |
|---|---|
| Too small | Progress is painfully slow; training may run out of time long before the bottom |
| About right | Steady progress to the minimum |
| Too large | Each step overshoots the valley and lands higher up the other side; the loss bounces or explodes |
There is no formula for the right value. Engineers try values spaced by factors of about 3 or 10 (0.1, 0.03, 0.01, 0.003) and watch the loss. Large models also change the learning rate during training with a schedule: a short warm-up from almost zero, then a slow decay, often following a cosine curve, so the final steps are small and careful.
Settings like this, which are chosen by people rather than learned, are called hyperparameters. The learning rate, the number of layers and k in k-nearest neighbours are all hyperparameters.
Batches, epochs and stochastic gradient descent
Calculating the gradient over the whole dataset for every single step is accurate but slow when there are millions of examples. Instead:
| Method | Gradient calculated from | Character |
|---|---|---|
| Batch gradient descent | Every example | Smooth, accurate, slow per step |
| Stochastic gradient descent (SGD) | One random example | Very noisy, very fast per step |
| Mini-batch gradient descent | A random batch, often 32 to 4,096 examples | The standard compromise, and what GPUs are built for |
One pass through the whole training set is an epoch. With 50,000 examples and a batch size of 100, one epoch is 500 steps.
The noise in mini-batch gradients is not only a cost. It jiggles the parameters out of narrow dips and flat regions and seems to help models generalise, which is one reason the method works as well as it does.
Watching training: loss curves
Engineers watch two curves while a model trains: the loss on the training data and the loss on the validation data.
- Both falling together: the model is still learning patterns that generalise.
- Training still falling but validation rising: the model has started to overfit, memorising the training examples.
- Early stopping keeps the parameters from the epoch with the lowest validation loss and stops there.
A training loss that suddenly shoots up usually means the learning rate is too high. A loss that will not fall at all can mean a bug, such as labels shuffled out of line with their examples. A validation loss lower than the training loss can happen early on, especially when training uses tricks such as dropout.
Valleys, saddles and plateaus in 3D
The bowl of linear regression has one bottom. The landscape of a neural network does not.
| Feature | What it is | Problem for descent |
|---|---|---|
| Local minimum | A dip that is not the lowest point overall | The ball can settle in it |
| Saddle point | A pass between hills: downhill in some directions, uphill in others | The gradient is near zero, so progress stalls |
| Plateau | A wide flat region | Tiny gradients, very slow progress |
| Ravine | A long narrow valley | Steps zigzag from wall to wall |
Two balls starting in different places can roll into different valleys, which is why the starting values matter. Surprisingly, in very high dimensions bad local minima are rarer than people feared: with millions of directions, there is almost always some direction still leading down. Saddles and plateaus cause more trouble.
Loss landscape lab
Roll balls down a rotating 3D loss surface with gradient descent, momentum or Adam. This tool needs JavaScript, so turn it on to use it. Everything you need to answer the exam question is explained in the text above.
Rotate the landscape in the lab above, drop balls in different places and change the learning rate and the optimiser. Watch plain gradient descent zigzag down the ravine, and momentum carry a ball through a shallow dip.
Momentum and Adam
Plain gradient descent forgets its past: each step depends only on the slope right here.
Momentum
Momentum gives the ball a memory, like a heavy ball rolling downhill. It keeps a running velocity: the gradients add up while they agree, so the ball speeds along a consistent slope, and zigzags cancel out across a ravine. It can even roll through small dips.
velocity ← 0.9 × velocity + gradient, then parameters ← parameters − learning rate × velocity
Adam
Adam (adaptive moment estimation, 2014, by Diederik Kingma and Jimmy Ba) combines momentum with a separate, automatically adjusted step size for every parameter: parameters whose gradients are consistently large take smaller steps, and rarely updated ones take larger steps. It is forgiving about the learning rate and is the default optimiser for most deep learning, including large language models, usually in a variant called AdamW.
Hands-on: gradient descent from scratch
Training the revision model
Level 4 found the best line with a formula: w ≈ 6.52, b ≈ 33.23. Gradient descent should find the same line without one.
hours = [1, 2, 2, 3, 4, 5, 6, 6, 7, 8]
score = [38, 45, 50, 52, 61, 64, 70, 75, 78, 86]
n = len(hours)
w, b = 0.0, 0.0
learning_rate = 0.03
for epoch in range(1, 1001):
dw = sum(2 * (w * x + b - y) * x for x, y in zip(hours, score)) / n
db = sum(2 * (w * x + b - y) for x, y in zip(hours, score)) / n
w -= learning_rate * dw
b -= learning_rate * db
if epoch in (1, 2, 5, 10, 50, 100, 500, 1000):
loss = sum((w * x + b - y) ** 2 for x, y in zip(hours, score)) / n
print(f"epoch {epoch:4}: w = {w:6.3f} b = {b:6.3f} loss = {loss:9.3f}")Too small, about right, too big
hours = [1, 2, 2, 3, 4, 5, 6, 6, 7, 8]
score = [38, 45, 50, 52, 61, 64, 70, 75, 78, 86]
n = len(hours)
def train(learning_rate, epochs):
w = b = 0.0
for _ in range(epochs):
dw = sum(2 * (w * x + b - y) * x for x, y in zip(hours, score)) / n
db = sum(2 * (w * x + b - y) for x, y in zip(hours, score)) / n
w, b = w - learning_rate * dw, b - learning_rate * db
if abs(w) > 1e6:
return "exploded!"
loss = sum((w * x + b - y) ** 2 for x, y in zip(hours, score)) / n
return f"w = {w:.2f}, b = {b:.2f}, loss = {loss:.2f}"
for lr in [0.001, 0.01, 0.03, 0.05]:
print(f"learning rate {lr:5}: after 500 epochs {train(lr, 500)}")Momentum through a ravine
The function f(x, y) = x² + 20y² is a long narrow valley. Compare plain descent with momentum, using the same learning rate and number of steps.
def grad(x, y):
return 2 * x, 40 * y
def descend(momentum, steps=60, lr=0.01):
x, y = -9.0, 1.0
vx = vy = 0.0
for _ in range(steps):
gx, gy = grad(x, y)
vx = momentum * vx + gx
vy = momentum * vy + gy
x, y = x - lr * vx, y - lr * vy
return x, y, x ** 2 + 20 * y ** 2
for m in [0.0, 0.5, 0.9]:
x, y, f = descend(m)
print(f"momentum {m}: ended at ({x:7.3f}, {y:7.3f}), loss {f:.5f}")Try this
- In the first program, start from w = 20, b = −50. Does it still find the same line?
- Find the largest learning rate that does not explode. The maths says it is just under 0.04 for this data.
- In the momentum program, try
lr=0.045. Plain descent now does well and momentum 0.9 overshoots. Why can too much momentum be a problem?
Level check
- Write the gradient descent update and explain each part
- Describe the effect of a learning rate that is too small and one that is too large
- Explain batch, stochastic and mini-batch gradient descent, and calculate steps per epoch
- Interpret training and validation loss curves, and explain early stopping
- Describe local minima, saddle points, plateaus and ravines
- Explain how momentum and Adam improve on plain gradient descent
Key terms you must be able to define
- Gradient descent
- Repeatedly moving the parameters a small step against the gradient of the loss.
- Learning rate
- The hyperparameter that sets how big each gradient descent step is.
- Hyperparameter
- A setting chosen by people rather than learned, such as the learning rate or the batch size.
- Epoch
- One complete pass through the training data.
- Mini-batch
- A small random subset of the training data used to estimate the gradient for one step.
- Stochastic gradient descent
- Gradient descent using the gradient from one example or a small random batch at each step.
- Early stopping
- Stopping training when the validation loss stops improving, and keeping the best parameters.
- Local minimum
- A point lower than everything near it but not the lowest point overall.
- Saddle point
- A point where the gradient is zero but the loss rises in some directions and falls in others.
- Momentum
- An optimiser that keeps a running velocity of past gradients, so steps build up along consistent slopes.
- Adam
- An optimiser combining momentum with an automatically adapted step size for each parameter; the usual default.
Flashcards
Go further: notes for future AI engineers
Why descent works in a billion dimensions
A neural network's loss landscape is not a bowl, and nothing guarantees that gradient descent finds the global minimum. It works in practice for several reasons researchers are still unpicking: in huge dimensions most critical points are saddles rather than traps; heavily over-parameterised networks have vast connected valleys of near-equally good solutions; and the noise of mini-batches steers training towards wide, flat valleys, whose solutions tend to generalise better than those in sharp, narrow ones.
Second-order methods
Gradient descent uses only the slope. Newton's method also uses the curvature (the second derivatives, collected in a matrix called the Hessian) to jump straight towards the bottom of a bowl. For a model with n parameters the Hessian has n² entries, which is impossible for a billion parameters, so deep learning uses cheap approximations of curvature instead. Adam's per-parameter step sizes are one such approximation.
Misconceptions to avoid
- Thinking a bigger learning rate always trains faster. Too big and the loss bounces or explodes.
- Confusing an epoch with a step. An epoch is a full pass through the data, which may be hundreds of steps.
- Stopping as soon as the training loss is low. Watch the validation loss, or the model may be overfitting.
- Assuming gradient descent always finds the best possible parameters. It finds a low point near where it started.
- Believing the noise in stochastic gradient descent is purely a problem. It also helps the model escape flat regions and generalise.
The idea of gradient descent was described by the French mathematician Augustin-Louis Cauchy in 1847, more than a century before the first neural network. It now runs on millions of processors, every second of every day.
Knowledge check
Check you could explain this
- LStart with random or initial values for the parameters
- LCalculate the gradient of the loss
- LUpdate each parameter by subtracting learning rate × gradient
- LRepeat until the loss stops decreasing / converges
Gradient descent starts with random or initial values for the parameters. It calculates the loss on the training data and the gradient of the loss with respect to each parameter. It then updates every parameter by subtracting the learning rate multiplied by its gradient, a small step downhill. These steps are repeated until the loss stops decreasing, when the parameters have converged near a minimum.
- LToo small: very small steps
- LSo training is very slow / may not reach the minimum in time
- LToo large: steps overshoot the minimum
- LSo the loss oscillates or diverges / explodes
If the learning rate is too small, every step is tiny, so training is very slow and may run out of time before reaching the minimum. If it is too large, each step overshoots the minimum and lands on the other side of the valley, so the loss oscillates and can even increase every step until it diverges, or explodes, to enormous values.
- LUp to epoch 20 the model learns patterns that generalise
- LAfter epoch 20 it is overfitting / memorising the training data
- LUse early stopping
- LKeep the parameters from epoch 20, the lowest validation loss
Until epoch 20 both losses fall, so the model is learning patterns that generalise to new data. After that the validation loss rises while the training loss keeps falling, which means the model is overfitting: memorising the training examples, including their noise. The engineer should use early stopping, keeping the parameters saved at epoch 20, where the validation loss was lowest, and could also add regularisation or more data.