Convolutions: How Machines See Edges and Shapes

Why not a plain dense network?

A modest 224 × 224 colour photo has 150,528 input numbers (Level 3). A dense first layer of just 1,000 neurons would need 150 million weights, before the network has learned anything. Worse, a dense layer ignores the structure of an image:

• It treats the pixel in the top-left corner and its neighbour as unrelated inputs, though neighbouring pixels belong together.
• A cat in the corner and a cat in the centre activate completely different weights, so it would have to learn "cat" separately for every position.

Two ideas fix both problems:

Together they give the convolution, the operation at the heart of a convolutional neural network (CNN).

The convolution operation

A kernel (or filter) is a small grid of weights, usually 3 × 3. It slides across the image. At each position:

1. Lay the kernel over a patch of the image.
2. Multiply each kernel weight by the pixel under it.
3. Add up the nine products. That sum is one number of the output.

Press Play to watch the kernel slide. The output grid, called a feature map, records how strongly the kernel's pattern appears at each position.

A worked patch

The sum is 27: a strong response. This kernel subtracts the left column from the right column, so it responds wherever the image gets brighter from left to right: a vertical edge. On a flat patch, all the same brightness, the left and right cancel and it outputs 0.

Kernels that find features

Different kernels detect different patterns.

For decades, computer vision engineers designed kernels like these by hand. The breakthrough of convolutional networks is that the kernel weights are parameters: they are learned by backpropagation, like any other weights. When researchers look inside trained networks, the first layer's kernels turn out to be edge detectors, colour blobs and stripe detectors, very like the ones people designed, plus many nobody had thought of.

Draw a shape in the lab, then try each kernel and watch which parts of your drawing light up in the feature map.

Stride, padding and the output size

Two settings control how the kernel moves.

• Stride: how many pixels the kernel jumps each time. Stride 1 visits every position; stride 2 skips every other one and halves the output size.
• Padding: a border of zeros added around the image, so the kernel can be centred on edge pixels too. "Same" padding keeps the output the same size as the input; "valid" (no padding) shrinks it.

Colour images and many filters

A colour image has three channels, so a kernel for it is a small block: 3 × 3 × 3, with 27 weights covering the red, green and blue values of a patch at once, plus a bias. Its output is still a single number per position, so one kernel produces one feature map.

A convolutional layer uses many kernels, often 32, 64 or more, each learning to detect a different pattern. Their feature maps are stacked into an output volume, which becomes the input to the next layer.

A layer of 64 filters of size 3 × 3 over a colour image has (3 × 3 × 3 + 1) × 64 = 1,792 parameters, whatever the size of the image. Compare that with the 150 million of the dense layer, and you see why convolution made computer vision practical.

Hands-on: convolution from scratch

Finding the edges of a bar

image = [[0] * 8 for _ in range(8)]
for row in range(1, 7):
    image[row][3] = image[row][4] = 9           # a bright vertical bar

def conv2d(img, kernel):
    n, k = len(img), len(kernel)
    out = []
    for r in range(n - k + 1):
        row = []
        for c in range(n - k + 1):
            row.append(sum(img[r + i][c + j] * kernel[i][j]
                           for i in range(k) for j in range(k)))
        out.append(row)
    return out

def show(grid):
    for row in grid:
        print(" ".join(f"{v:4}" for v in row))

vertical = [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]
horizontal = [[-1, -2, -1], [0, 0, 0], [1, 2, 1]]
print("Vertical edges:")
show(conv2d(image, vertical))
print("Horizontal edges:")
show(conv2d(image, horizontal))

The vertical edge kernel lights up down both sides of the bar: positive where the image gets brighter to the right, negative where it gets darker. The horizontal kernel only fires at the bar's top and bottom ends.

Output sizes and parameter counts

def output_size(n, k, padding=0, stride=1):
    return (n + 2 * padding - k) // stride + 1

print("6x6, 3x3 kernel, no padding:", output_size(6, 3))
print("6x6, 3x3 kernel, padding 1:", output_size(6, 3, padding=1))
print("224x224, 7x7 kernel, padding 3, stride 2:", output_size(224, 7, 3, 2))

def conv_params(k, channels_in, filters):
    return (k * k * channels_in + 1) * filters

def dense_params(inputs, neurons):
    return inputs * neurons + neurons

print(f"Conv layer, 64 filters of 3x3 on RGB: {conv_params(3, 3, 64):,} parameters")
print(f"Dense layer, 224x224x3 to 1,000 neurons: {dense_params(224 * 224 * 3, 1000):,} parameters")

Try this

1. Add a blur kernel, with every weight 1/9, and run it on the bar. What happens to the bar's edges?
2. Make the bar diagonal instead. Which kernel responds more strongly now?
3. Design a 3 × 3 kernel that responds to a single bright pixel surrounded by dark ones.

Level check

• Explain two problems with using dense layers for images, and how local connections and weight sharing fix them
• Calculate one output value of a convolution by hand
• Describe what edge, blur and sharpen kernels do, and explain how CNNs get their kernels
• Use the formula to work out the output size for a given stride and padding
• Explain how kernels work on colour images and what a layer of many filters produces
• Calculate the number of parameters in a convolutional layer

Key terms you must be able to define

Convolution

Sliding a small kernel across an image and taking the weighted sum of each patch it covers.

Kernel (filter)

A small grid of weights, such as 3 × 3, that detects one pattern wherever it appears.

Feature map

The grid of outputs produced by one kernel, showing where its pattern appears.

Weight sharing

Using the same kernel weights at every position, so a pattern learned once is found everywhere.

Local connection

Each output depends only on a small patch of the input.

Stride

How many pixels the kernel moves between positions.

Padding

A border of zeros added around the input so the kernel can cover the edges.

Channel

One layer of an input or output volume, such as red, or one feature map.

Convolutional neural network (CNN)

A network built mainly from convolutional layers, the standard architecture for images.

Edge detector

A kernel that responds to sudden changes in brightness, such as the Sobel kernels.

Flashcards

Go further: notes for future AI engineers

Convolution or cross-correlation?

Strictly, the operation in this lesson is cross-correlation: a true mathematical convolution flips the kernel first. Because a CNN learns its kernels, the flip makes no difference (it would simply learn the flipped kernel), so deep learning libraries implement cross-correlation and call it convolution. It is worth knowing when you read signal-processing texts.

Translation equivariance

Because the same kernel is applied everywhere, shifting the input shifts the feature map by the same amount: convolution is translation equivariant. That is the built-in assumption, the inductive bias, that makes CNNs so data-efficient for images. Pooling and the final layers then make the overall answer roughly invariant to position: a cat is a cat wherever it sits. Vision transformers (Level 10) drop this assumption and must learn it from much more data.

Misconceptions to avoid