Activation Functions: The Bend That Makes Networks Powerful

Why a neuron needs a bend

Remove the activation function and each layer just multiplies by a matrix and adds a vector. Stack two such layers:

However many linear layers you stack, the result is still one linear function: one straight line, one flat plane. Depth would be pointless.

An activation function puts a bend (a non-linearity) into every neuron. Bends can be combined into curves, and curves can be combined into any shape. That is where a neural network's power comes from.

The activation functions you will meet

ReLU: the bend that made deep learning work

The rectified linear unit, ReLU(z) = max(0, z), became the standard around 2010 to 2012, and AlexNet's success in 2012 relied on it. It has three big advantages:

• Cheap: one comparison, no exponentials.
• Strong gradients: for any positive z its slope is exactly 1, so the learning signal passes through many layers without shrinking.
• Sparse: for negative z it outputs exactly 0, so only some neurons are active for a given input, which makes networks efficient.

Its weakness: a neuron whose z is negative for every input outputs 0 and has slope 0 everywhere, so it can never recover. These are dead neurons. Leaky ReLU keeps a small slope for negative z to fix this, and smooth variants such as GELU and SiLU (also called Swish) work slightly better in very large models.

Building curves from ReLUs

Each ReLU neuron adds one "hinge": a point where the line is allowed to change direction. Add several, each with its own weight and bias, and you can trace almost any curve.

This is the idea behind the universal approximation theorem: a network with one hidden layer of enough neurons can approximate any continuous function as closely as you like. Each hidden neuron contributes one bend, and the output layer adds them up with weights.

The theorem says such a network exists. It does not say gradient descent will find it, or how many neurons it needs. In practice, deep networks, with many layers each folding the space again, need far fewer neurons than one enormous shallow layer.

Derivatives and vanishing gradients

Training needs each activation's slope (Level 5), and the slopes multiply together as the error signal passes back through the layers (Level 7).

The sigmoid's slope is σ(z) × (1 − σ(z)), which is never more than 0.25. Pass a gradient back through ten sigmoid layers and it is multiplied by at most 0.25 ten times:

This vanishing gradient problem is a big reason deep networks were so hard to train before about 2010. ReLU's slope of 1 for positive inputs, together with better starting weights, normalisation layers and the shortcut connections of ResNets (Level 8), is what made networks with dozens and then hundreds of layers trainable.

Softmax: scores into probabilities

A classifier choosing among many classes ends with one output per class, a raw score called a logit. Softmax turns the logits into probabilities:

• Every output is between 0 and 1, and they add up to exactly 1.
• The order is kept: the biggest logit gets the biggest probability.
• Exponentials exaggerate differences: a logit 2 higher gives about 7.4 times the probability.

Temperature

Dividing the logits by a temperature T before softmax controls how peaked the result is. T below 1 sharpens it towards the top choice; T above 1 flattens it towards an even spread. Language models use exactly this to decide how adventurous their word choices are (Level 11).

Choosing activations

Hands-on: activations in code

The functions, side by side

import math

def sigmoid(z): return 1 / (1 + math.exp(-z))
def tanh(z): return math.tanh(z)
def relu(z): return max(0.0, z)
def leaky_relu(z): return z if z > 0 else 0.01 * z
def gelu(z): return 0.5 * z * (1 + math.erf(z / math.sqrt(2)))

print("   z  sigmoid   tanh   relu  leaky   gelu")
for z in [-3, -1, -0.5, 0, 0.5, 1, 3]:
    print(f"{z:4}  {sigmoid(z):7.3f} {tanh(z):6.3f} {relu(z):6.2f} {leaky_relu(z):6.2f} {gelu(z):6.3f}")

Vanishing gradients

import math

def sigmoid(z): return 1 / (1 + math.exp(-z))
def sigmoid_slope(z): return sigmoid(z) * (1 - sigmoid(z))

gradient = 1.0
for layer in range(1, 11):
    gradient *= sigmoid_slope(0)          # 0.25, the best case
    print(f"after {layer:2} sigmoid layers: {gradient:.8f}")
print("through 10 ReLU layers with positive inputs:", 1.0 ** 10)

Softmax and temperature

import math

def softmax(logits, temperature=1.0):
    exps = [math.exp(z / temperature) for z in logits]
    total = sum(exps)
    return [e / total for e in exps]

classes = ["cat", "dog", "fox", "owl"]
logits = [2.0, 1.0, 0.1, -1.0]
for T in [0.5, 1.0, 2.0]:
    probs = softmax(logits, T)
    shown = ", ".join(f"{c} {p:.2f}" for c, p in zip(classes, probs))
    print(f"T = {T}: {shown}   (sum {sum(probs):.2f})")

A bump built from three ReLUs

def relu(z): return max(0.0, z)

def bump(x):
    # three hidden neurons, then a weighted sum: up from 1, down from 2, flat after 3
    return 1.0 * relu(x - 1) - 2.0 * relu(x - 2) + 1.0 * relu(x - 3)

for i in range(9):
    x = i * 0.5
    print(f"x = {x:3}: {bump(x):4.1f} " + "#" * int(bump(x) * 20))

Try this

1. Add a fourth ReLU to the bump so that it rises again after x = 3.5.
2. What happens to softmax at T = 0.1? And at T = 100?
3. Find the z at which the sigmoid's slope is largest. Why does it make sense that it is there?

Level check

• Show that stacking linear layers without activations gives a single linear function
• Sketch the sigmoid, tanh, ReLU and leaky ReLU functions and state their ranges
• Give three reasons ReLU became the default activation for hidden layers
• Explain how a sum of ReLUs can approximate a curve, and what the universal approximation theorem says
• Explain the vanishing gradient problem using the sigmoid's maximum slope
• Calculate softmax for a few logits and explain the effect of temperature

Key terms you must be able to define

Non-linearity

A bend in a function. Activation functions add one to every neuron so that layers do not collapse into one.

Sigmoid

1 ÷ (1 + e^-z), squashing any number into the range 0 to 1.

Tanh

The hyperbolic tangent, an S-shaped function from −1 to 1, centred on zero.

ReLU

The rectified linear unit, max(0, z): the default activation for hidden layers.

Leaky ReLU

Like ReLU but with a small slope for negative inputs, so neurons cannot die.

GELU

A smooth ReLU-like activation used in transformers.

Dead neuron

A ReLU neuron whose input is negative for every example, so it outputs 0 and never learns.

Universal approximation theorem

A network with one hidden layer of enough neurons can approximate any continuous function.

Vanishing gradient

A learning signal that shrinks layer by layer until the early layers barely learn.

Logit

A raw output score before it is turned into a probability.

Softmax

A function turning a list of logits into probabilities that are all positive and add up to 1.

Temperature

A number the logits are divided by before softmax; low values sharpen the probabilities, high values flatten them.

Flashcards

Go further: notes for future AI engineers

Numerical stability in softmax

e to the power 1,000 overflows a computer's floating point numbers. Real implementations subtract the largest logit from every logit before exponentiating: softmax(z) = softmax(z − max(z)), which gives exactly the same probabilities because the common factor cancels. Libraries also combine softmax with the logarithm in cross-entropy (the log-softmax) to avoid ever computing tiny probabilities directly. If you implement these yourself and see NaN, "not a number", in your loss, this is usually why.

Why the slope of 1 matters so much

In a network of depth L, the gradient reaching the first layer is roughly a product of L factors, one per layer. If the typical factor is a little below 1 the product vanishes; a little above 1 and it explodes. Good initialisation schemes (Xavier for tanh, He for ReLU) choose the starting weights' spread so the typical factor is close to 1, layer after layer. Residual connections (Level 8) add a path where the factor is exactly 1, which is why networks hundreds of layers deep can train at all.

Misconceptions to avoid