Convolutional Networks: From Pixels to Predictions

Pooling: keeping the strongest signal

After a convolution and ReLU, a pooling layer shrinks each feature map. Max pooling with a 2 × 2 window and stride 2 keeps only the largest value in each 2 × 2 block.

Pooling has no parameters to learn. Average pooling takes the mean instead of the maximum, and global average pooling averages each whole feature map down to a single number, which modern networks use just before the final layer.

The CNN architecture

A classic CNN repeats a pattern of convolution, ReLU and pooling, then finishes with dense layers:

1. Convolution + ReLU: detect features.
2. Pooling: shrink the maps, keeping the strongest signals.
3. Repeat, with more filters each time.
4. Flatten the final volume into one long vector.
5. Dense layers combine the features, ending in softmax over the classes.

Two trends run through the network: the spatial size shrinks (32 × 32, then 16 × 16, then 8 × 8) while the number of channels grows (3, then 16, then 32). The network trades "where" for "what": fewer positions, but richer descriptions of each.

What each layer learns

Each value in a deeper layer depends on a larger patch of the original image, called its receptive field. Two stacked 3 × 3 convolutions see a 5 × 5 patch; three see 7 × 7; pooling doubles the growth.

So deeper layers can detect larger and more complex things. Researchers who visualised trained networks (Matthew Zeiler and Rob Fergus, 2013) found a clear hierarchy:

No one told the network to look for eyes or wheels. Those detectors emerged because they helped reduce the loss.

Famous architectures

Residual connections

Before 2015, adding more layers past about 20 made networks worse, even on their training data: the gradients could not get through. The ResNet team, led by Kaiming He, added a shortcut that carries a block's input straight past it and adds it to the output:

The block now only has to learn the change F(x) to make to its input, and if no change helps it can push F towards zero and pass x through untouched. During backpropagation, the gradient flows straight back along the shortcut, so even the earliest layers receive a strong learning signal. Residual connections are now everywhere, including inside every transformer.

Making vision models work in practice

Data augmentation

Show the network modified copies of each training image: flipped, cropped, rotated slightly, with brightness and colour changed. The label stays the same, so the network learns that these changes do not matter, and it sees far more variety than the dataset contains.

Other essentials

• Dropout: during training, randomly switch off a fraction of neurons (often half) at each step, so the network cannot rely on any single one. It is a strong defence against overfitting.
• Batch normalisation: rescale each layer's outputs to a steady mean and spread during training, which makes deep networks train faster and more reliably.
• Transfer learning: start from a network already trained on ImageNet, keep its convolutional layers as a feature extractor, and train only a new final layer on your own small dataset. A few hundred images can then be enough.

Beyond "what is in the picture?"

Hands-on: a hand-built CNN

Max pooling

def max_pool(grid, size=2):
    out = []
    for r in range(0, len(grid), size):
        row = []
        for c in range(0, len(grid[0]), size):
            row.append(max(grid[r + i][c + j] for i in range(size) for j in range(size)))
        out.append(row)
    return out

feature_map = [[1, 3, 0, 2],
               [4, 6, 1, 0],
               [0, 2, 8, 5],
               [1, 1, 3, 7]]
print(max_pool(feature_map))

Line detector: convolution, ReLU, pooling, decision

This tiny CNN has two fixed kernels, one for horizontal lines and one for vertical lines. It convolves, applies ReLU, takes the global maximum of each feature map, and picks the larger.

def conv2d(img, k):
    n = len(img)
    return [[sum(img[r + i][c + j] * k[i][j] for i in range(3) for j in range(3))
             for c in range(n - 2)] for r in range(n - 2)]

def relu_max(grid):
    return max(max(0, v) for row in grid for v in row)

horizontal = [[-1, -1, -1], [2, 2, 2], [-1, -1, -1]]
vertical = [[-1, 2, -1], [-1, 2, -1], [-1, 2, -1]]

def blank():
    return [[0] * 6 for _ in range(6)]

h_line, v_line, cross = blank(), blank(), blank()
for i in range(1, 5):
    h_line[2][i] = 1
    v_line[i][3] = 1
    cross[2][i] = cross[i][2] = 1

for name, img in [("horizontal line", h_line), ("vertical line", v_line), ("cross", cross)]:
    h = relu_max(conv2d(img, horizontal))
    v = relu_max(conv2d(img, vertical))
    verdict = "horizontal" if h > v else "vertical" if v > h else "both equally"
    print(f"{name:15}: horizontal score {h}, vertical score {v} -> {verdict}")

A real CNN does exactly this, except that it learns dozens of kernels per layer instead of using two we chose, stacks many layers, and learns the final decision too.

Data augmentation by flipping

image = [[1, 2, 3],
         [4, 5, 6],
         [7, 8, 9]]
flipped = [row[::-1] for row in image]          # mirror left to right
rotated = [list(row) for row in zip(*image[::-1])]  # rotate 90 degrees clockwise
print("original:", image)
print("flipped: ", flipped)
print("rotated: ", rotated)

Try this

1. Add a diagonal line image. What do the two detectors say? Design a third kernel for diagonals.
2. Pool the feature map with size 4. What single number do you get, and what does it mean?

Level check

• Explain max pooling and give three benefits of pooling
• Describe the layers of a classic CNN and how the shape of the data changes through it
• Explain receptive fields and the hierarchy of features a CNN learns
• Name three famous CNN architectures and the key idea each introduced
• Explain how residual connections work and why they let networks go deeper
• Describe data augmentation, dropout and transfer learning

Key terms you must be able to define

Pooling

Shrinking a feature map by summarising each small block, usually by its maximum.

Max pooling

Keeping only the largest value in each window, such as each 2 × 2 block.

Flatten

Turning a volume of feature maps into one long vector for the dense layers.

Receptive field

The patch of the original image that one value in a layer depends on.

Feature hierarchy

Simple features in early layers combining into complex ones in later layers.

Residual connection

A shortcut that adds a block's input to its output, so the block learns only the change: F(x) + x.

Data augmentation

Training on modified copies of the data, such as flipped or cropped images, to add variety.

Dropout

Randomly switching off neurons during training so the network cannot rely on any one of them.

Batch normalisation

Rescaling a layer's outputs during training to keep their mean and spread steady.

Object detection

Finding every object in an image, with a bounding box and a label for each.

Segmentation

Labelling every pixel of an image with the class it belongs to.

Flashcards

Go further: notes for future AI engineers

Why residual connections fix the gradient

In a plain deep network the gradient reaching layer 1 is a product of one factor per layer, so it vanishes or explodes. With a residual block y = F(x) + x, the derivative of y with respect to x is F′(x) + 1. The "+ 1" means that even if F′ is tiny, the gradient passes back almost unchanged along the identity path, and stacking blocks adds paths of every length instead of multiplying one long chain. A ResNet behaves somewhat like an ensemble of many shallower networks.

The cost of convolution

A convolutional layer's parameter count is small, but its compute is not: a 3 × 3 kernel with 64 input and 64 output channels on a 112 × 112 map needs about 3 × 3 × 64 × 64 × 112 × 112, roughly 460 million multiply-adds, for one image. Efficient architectures such as MobileNet split it into a cheap per-channel convolution and a 1 × 1 convolution that mixes channels (a depthwise separable convolution), cutting the cost about eightfold, which is how vision models run on phones.

Misconceptions to avoid