Backpropagation: How a Network Learns

The credit assignment problem

A network makes a wrong prediction. It has a million weights. Which ones were to blame, and by how much should each change?

Gradient descent (Level 5) answers in principle: move every weight against its partial derivative of the loss. The question is how to get a million partial derivatives cheaply. Nudging each weight in turn (Level 5) would take a million and more forward passes per step.

Backpropagation gets every one of them in a single backward sweep through the network, costing about twice a forward pass. It is the reason deep learning is possible.

The chain rule, in gears

If y depends on u, and u depends on x, then a small change in x passes through u to y:

Think of gears. If gear B turns 3 times for every turn of gear A, and gear C turns 2 times for every turn of B, then C turns 3 × 2 = 6 times for every turn of A. Rates of change multiply along a chain. A neural network is a long chain of simple steps, so the effect of any weight on the loss is a product of local rates of change along the path from that weight to the loss.

Computational graphs: forward then backward

Write any calculation as a computational graph: a diagram of simple operations, each node taking inputs and producing an output.

The example computes the squared error of a one-weight model, L = (w × x + b − y)², with x = 2, w = 0.5, b = 0.1 and target y = 1.

1. Forward pass, left to right: compute each node's value. u = w × x = 1, z = u + b = 1.1, e = z − y = 0.1, L = e² = 0.01.
2. Backward pass, right to left: start with dL/dL = 1, and at each node multiply the incoming gradient by the node's local derivative.

So dL/dw = 0.4 and dL/db = 0.2. Every node only needs to know its own simple derivative; the chain rule does the rest. That locality is what makes backpropagation easy to program and fast to run.

Through a neuron, with real numbers

Now add a sigmoid: z = w × x + b, a = σ(z), L = (a − y)². With x = 2, w = 0.5, b = 0.1, y = 1:

The sigmoid's local derivative is a(1 − a) = 0.7503 × 0.2497 = 0.1874. Then dL/dw = dL/dz × x = −0.1872 and dL/db = dL/dz × 1 = −0.0936.

Both gradients are negative, so gradient descent increases w and b. With a learning rate of 0.5: w becomes 0.5936 and b becomes 0.1468. Run forward again: a = 0.7915 and the loss falls from 0.0624 to 0.0435. One step of learning, done by hand.

Backpropagation through a whole network

For a deep network the same process runs layer by layer, from the output back to the input:

1. Forward pass: compute and remember every layer's activations.
2. Compute the loss and its gradient with respect to the output.
3. For each layer, from last to first, turn the gradient arriving at its outputs into gradients for its weights, its biases and its inputs.
4. Update every weight with gradient descent.

Inside each layer, step 3 does three small things:

• multiply the incoming gradient by the activation's derivative, the local slope;
• combine that with the layer's inputs to get the gradients of its weights and biases;
• multiply by the layer's weights, transposed, to get the gradient to pass to the layer below.

Each layer receives "how much the loss would change if my outputs changed" and passes back "how much the loss would change if my inputs changed". That is why it is called back propagation of errors.

The training loop

Forward pass, loss, backward pass, update, repeated for every batch and every epoch. In a framework such as PyTorch, the backward pass is a single call, because the framework records the computational graph during the forward pass automatically:

import torch

model = torch.nn.Sequential(torch.nn.Linear(2, 3), torch.nn.Tanh(), torch.nn.Linear(3, 1))
optimiser = torch.optim.SGD(model.parameters(), lr=0.5)
loss_fn = torch.nn.BCEWithLogitsLoss()

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

for epoch in range(2000):
    loss = loss_fn(model(X), y)     # forward pass and loss
    optimiser.zero_grad()
    loss.backward()                 # backpropagation: every gradient at once
    optimiser.step()                # gradient descent update

This is called automatic differentiation, or autograd. The hands-on section builds a tiny version from scratch.

When gradients misbehave

Backpropagation multiplies many local derivatives together, so the gradient can shrink or grow exponentially with depth.

Use the playground above to watch backpropagation shape a network's decision boundary in real time. Try a spiral with one hidden neuron, then with eight, then with two layers.

Hands-on: backpropagation from scratch

A tiny automatic differentiation engine

Each Value remembers how it was made. Calling backward() walks the graph in reverse and applies the chain rule at every node.

import math

class Value:
    def __init__(self, data, parents=()):
        self.data, self.grad = data, 0.0
        self.parents = parents
        self.backward_step = lambda: None

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other))
        def step():
            self.grad += out.grad
            other.grad += out.grad
        out.backward_step = step
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other))
        def step():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out.backward_step = step
        return out

    def __sub__(self, other):
        return self + (-1 * other if isinstance(other, Value) else Value(-other))

    def __rmul__(self, other):
        return self * other

    def sigmoid(self):
        s = 1 / (1 + math.exp(-self.data))
        out = Value(s, (self,))
        def step():
            self.grad += s * (1 - s) * out.grad
        out.backward_step = step
        return out

    def backward(self):
        order, seen = [], set()
        def visit(v):
            if id(v) not in seen:
                seen.add(id(v))
                for p in v.parents:
                    visit(p)
                order.append(v)
        visit(self)
        self.grad = 1.0
        for v in reversed(order):
            v.backward_step()

# the worked example: z = w*x + b, a = sigmoid(z), L = (a - y)^2
w, b = Value(0.5), Value(0.1)
a = (w * 2 + b).sigmoid()
error = a - 1
L = error * error
L.backward()
print(f"loss {L.data:.4f}, dL/dw = {w.grad:.4f}, dL/db = {b.grad:.4f}")

The engine reproduces the hand calculation: dL/dw = −0.1872 and dL/db = −0.0936.

Training a network to learn XOR

Now the real thing: a network with 2 inputs, 3 tanh hidden neurons and a sigmoid output, trained by backpropagation and gradient descent, written out by hand.

import math, random

random.seed(2)
H = 3
w1 = [[random.uniform(-1, 1) for _ in range(2)] for _ in range(H)]
b1 = [random.uniform(-1, 1) for _ in range(H)]
w2 = [random.uniform(-1, 1) for _ in range(H)]
b2 = 0.0
data = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]
lr = 0.5

def forward(x):
    h = [math.tanh(w1[j][0] * x[0] + w1[j][1] * x[1] + b1[j]) for j in range(H)]
    p = 1 / (1 + math.exp(-(sum(w2[j] * h[j] for j in range(H)) + b2)))
    return h, p

for epoch in range(1, 1001):
    loss = 0.0
    for x, y in data:
        h, p = forward(x)                                   # forward pass
        loss -= y * math.log(p) + (1 - y) * math.log(1 - p)
        d_out = p - y                                       # gradient at the output
        for j in range(H):
            d_h = d_out * w2[j] * (1 - h[j] ** 2)           # back through w2 and tanh
            w2[j] -= lr * d_out * h[j]                      # update output weights
            w1[j][0] -= lr * d_h * x[0]                     # update hidden weights
            w1[j][1] -= lr * d_h * x[1]
            b1[j] -= lr * d_h
        b2 -= lr * d_out
    if epoch in (1, 50, 100, 200, 500, 1000):
        print(f"epoch {epoch:4}: average loss {loss / 4:.4f}")

for x, y in data:
    print(x, "->", round(forward(x)[1], 3), " target", y)

With sigmoid outputs and cross-entropy loss, the gradient at the output is simply p − y, one of the neat results that makes this pairing so popular.

Try this

1. Change random.seed(2) to random.seed(3). Training gets stuck with every output near 0.5. A bad start can trap a network on a flat region; try H = 4 with the same seed.
2. Set lr = 5. What happens to the loss?
3. Use the Value engine to check one of the XOR network's gradients.

Level check

• Explain the credit assignment problem and why nudging each weight is too slow
• State the chain rule and explain it with an example
• Carry out a forward and backward pass on a small computational graph
• Calculate the gradients of a single sigmoid neuron and one gradient descent update
• Describe how backpropagation passes gradients through the layers of a network
• Describe vanishing and exploding gradients and why weights start random

Key terms you must be able to define

Backpropagation

Computing the gradient of the loss with respect to every weight by applying the chain rule backwards through the network.

Credit assignment problem

Working out how much each weight contributed to an error.

Chain rule

The rate of change through a chain of functions is the product of each link's rate of change.

Computational graph

A calculation drawn as a network of simple operations, used for forward and backward passes.

Local derivative

The derivative of one node's output with respect to its inputs, which is all each node needs for backpropagation.

Automatic differentiation

Software that records a calculation and computes its derivatives automatically; also called autograd.

Training loop

Forward pass, loss, backward pass and update, repeated over batches and epochs.

Exploding gradient

A gradient that grows exponentially through the layers, making training unstable.

Gradient clipping

Capping the size of a gradient to stop exploding gradients.

Weight initialisation

Choosing random starting weights, scaled to each layer's size, so that every neuron can learn something different.

Flashcards

Go further: notes for future AI engineers

Reverse mode versus forward mode

There are two ways to apply the chain rule to a graph. Forward mode pushes derivatives from one input to all the outputs; reverse mode pulls derivatives from one output back to all the inputs. A network has millions of inputs (its weights) and one output (the loss), so reverse mode gets every gradient in one sweep, while forward mode would need one sweep per weight. Backpropagation is reverse-mode automatic differentiation.

Memory is the price

Reverse mode needs the activations from the forward pass, so training stores every layer's activations for the whole batch. That is often what limits batch size and model size on a GPU. Gradient checkpointing saves memory by storing only some activations and recomputing the rest during the backward pass, trading extra compute for less memory, and is standard practice for training large models.

Misconceptions to avoid