Neural Networks: Layers, Matrices and the Forward Pass

From one neuron to a network

A neural network connects neurons in layers. Every neuron in one layer feeds every neuron in the next, which is called a fully connected or dense layer.

A network with at least one hidden layer is often called a multi-layer perceptron (MLP). "Deep" learning simply means many hidden layers. The values computed by the hidden neurons for an input are called its activations.

A layer is a matrix multiplication

Each neuron in a layer computes a weighted sum of the same inputs, with its own weights. Stack each neuron's weights as a row of a weight matrix W, and the whole layer is one calculation:

To multiply a matrix by a vector, take the dot product of each row with the vector. Each row gives one neuron's weighted sum. A layer with 3 inputs and 4 neurons has a 4 × 3 weight matrix and a bias vector of 4 numbers.

Batches

Feed a batch of examples at once by stacking them as the rows of a matrix X. The layer becomes X Wᵀ + b, and one matrix multiplication handles every example in the batch together.

The forward pass, worked through

The forward pass carries an input through the network, layer by layer, to the output. Here is a tiny network that solves XOR with ReLU hidden neurons:

Four inputs, four correct answers: what a single neuron could never do.

What hidden layers really do

The hidden layer transforms the space. It moves each input point to a new position, given by the hidden neurons' outputs, where the problem becomes easy.

In the input space the XOR classes sit on opposite diagonals and no line separates them. In the hidden space (h₁, h₂), the points (0, 1) and (1, 0) land on the same spot, and a single straight line now divides the classes. The output neuron only has to draw that line.

This is the deep idea of deep learning. Each layer re-represents the data so that the next layer's job is simpler. In an image network, early layers turn pixels into edges, later layers turn edges into textures and shapes, and the last layers turn shapes into "cat" or "aeroplane". Nobody programs these features: training discovers them.

Width and depth

The universal approximation theorem (Level 6) says one wide hidden layer is enough in principle. In practice, depth is far more efficient: each layer can fold and reuse what the previous one built, so a deep network can represent with thousands of neurons what a shallow one would need millions for.

Counting parameters

A dense layer from n inputs to m neurons has:

A classic network for handwritten digits, with 784 inputs, hidden layers of 128 and 64 neurons and 10 outputs:

Stored as 32-bit floating point numbers (4 bytes each), that network takes about 0.44 MB. A model with 70 billion parameters takes 280 GB in that format, which is why large models are stored in 16-bit or smaller formats (Level 14).

Hands-on: a network from lists

A dense layer as matrix times vector

def matvec(W, x):
    return [sum(w * xi for w, xi in zip(row, x)) for row in W]

def dense(W, b, x, activation):
    return [activation(z + bias) for z, bias in zip(matvec(W, x), b)]

def relu(z): return max(0.0, z)
def identity(z): return z

# the XOR network from the lesson
W1, b1 = [[1, 1], [1, 1]], [0, -1]
W2, b2 = [[1, -2]], [0]

for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h = dense(W1, b1, x, relu)
    out = dense(W2, b2, h, identity)
    print(f"x = {x} -> hidden {h} -> output {out[0]}")

Counting parameters

def count_parameters(sizes):
    total = 0
    for n_in, n_out in zip(sizes, sizes[1:]):
        layer = n_in * n_out + n_out
        print(f"  {n_in:>5} -> {n_out:<5}: {layer:>11,} parameters")
        total += layer
    return total

for sizes in [[784, 128, 64, 10], [784, 512, 512, 10], [2, 3, 1]]:
    total = count_parameters(sizes)
    print(f"{sizes}: {total:,} parameters, {total * 4 / 1e6:.2f} MB as 32-bit numbers\n")

A forward pass on a batch

def matmul(A, B):
    # (rows of A) x (columns of B)
    return [[sum(a * b for a, b in zip(row, col)) for col in zip(*B)] for row in A]

X = [[0, 0], [0, 1], [1, 0], [1, 1]]          # a batch of four examples
W1_T = [[1, 1], [1, 1]]                       # weights, one column per hidden neuron
hidden = [[max(0, z + b) for z, b in zip(row, [0, -1])] for row in matmul(X, W1_T)]
print("hidden activations for the whole batch:", hidden)

Try this

1. Change the output weights in the XOR network from [1, −2] to [1, −1]. Which input now gets the wrong answer?
2. How many parameters does a network with sizes [3, 4, 4, 1] have? Check with the code.

Level check

• Describe input, hidden and output layers, and what makes a layer dense
• Write a dense layer as output = f(Wx + b) and give the shape of W
• Carry out a forward pass through a small network by hand
• Explain how a hidden layer makes XOR linearly separable
• Explain why depth helps, with the idea of features built from features
• Count the weights and biases in a network from its layer sizes

Key terms you must be able to define

Neural network

Layers of artificial neurons, each layer feeding the next.

Dense layer

A layer in which every neuron receives every output of the previous layer; also called fully connected.

Hidden layer

A layer between input and output whose values are internal to the network.

Multi-layer perceptron (MLP)

A network of dense layers with at least one hidden layer.

Weight matrix

The weights of a whole layer, one row per neuron and one column per input.

Forward pass

Computing a network's output from an input, layer by layer.

Activations

The outputs of a layer's neurons for a particular input.

Representation

The way data is encoded at some layer; each layer re-represents the data for the next.

Parameter count

The total number of weights and biases, which sets a model's size and memory.

Deep learning

Machine learning with neural networks that have many hidden layers.

Flashcards

Go further: notes for future AI engineers

Tensors, shapes and broadcasting

Real frameworks work with whole tensors. A batch of 32 examples with 784 features is a (32, 784) tensor; multiplying by a (784, 128) weight matrix gives (32, 128); adding a bias of shape (128,) is broadcast across all 32 rows automatically. Most bugs in deep learning code are shape mistakes, so engineers write the expected shape beside every line, and check it.

The cost of a forward pass

Multiplying an (n × k) matrix by a (k × m) one takes about n × k × m multiplications and as many additions, so roughly 2 × parameters floating point operations (FLOPs) per example for a dense network. A training step costs about three times a forward pass, because the backward pass needs about twice as much work. Level 11 uses exactly this rule of thumb to estimate what it costs to train a large language model.

Misconceptions to avoid