Autoencoders and GANs: Machines That Create

Telling apart, or making new

Most of this course so far has been about discriminative models: given an input, predict a label. A generative model learns what the data itself looks like, well enough to produce new examples that could have come from it.

A language model (Level 11) is a generative model of text: it learns the probability of every sequence of tokens and samples new ones. This level is about generating images, sound and more, where the data is a grid of numbers rather than a sequence of tokens.

Autoencoders: squeeze, then rebuild

An autoencoder is two networks back to back. The encoder squeezes the input into a much smaller vector, the latent code; the decoder tries to rebuild the original from that code alone. Training minimises the reconstruction error between input and output.

A 28 × 28 image has 784 numbers; squeezed through a bottleneck of 32, the network cannot copy the image, so it must learn which features matter most: the slant of a stroke, the width of a loop. Uses include:

• Compression: store or send the small code instead of the image.
• Denoising: train with noisy inputs and clean targets, and the network learns to remove noise.
• Anomaly detection: an autoencoder trained only on normal data rebuilds unusual data badly, so a large reconstruction error flags something odd, such as a faulty machine part.

Latent space

The latent codes form a latent space. Because the decoder must rebuild similar inputs from codes, similar inputs end up with nearby codes, just as similar words had nearby embeddings in Level 9.

Walking in a straight line between two codes and decoding each step morphs one output into another. Moving in a particular direction can change one property at a time: in a model of faces, one direction might add a smile and another add glasses.

Variational autoencoders

A plain autoencoder's latent space has gaps: decode a random point between the codes it learned and you may get nonsense. A variational autoencoder (VAE), introduced in 2013, fixes this so that you can sample new data:

1. The encoder outputs a mean and a spread for each latent number, describing a small cloud rather than a single point.
2. A code is sampled from that cloud: z = mean + spread × a random number.
3. The decoder rebuilds the input from z.
4. The loss adds a second term that pulls every cloud towards a standard bell curve centred on zero.

That second term packs the clouds together into one smooth region with no gaps. To create something new, pick a random point from the standard bell curve and decode it.

Generative adversarial networks

A generative adversarial network (GAN), from 2014, trains two networks against each other:

• The generator turns random noise into fake data. It never sees a real example.
• The discriminator is shown real examples and fakes, and learns to tell them apart.

It is a contest between a forger and a detective. Each time the detective gets better at spotting fakes, the gradient of its verdict tells the forger exactly how to change its fakes to be more convincing. They take turns: a few steps training the discriminator, then a few training the generator to make the discriminator answer "real".

In theory the contest ends when the generator's fakes are distributed exactly like the real data. Then the best the discriminator can do is guess, and it outputs 0.5 for everything.

Why GANs are hard to train

Two networks chasing each other do not always settle down:

• Mode collapse: the generator finds a few outputs that fool the discriminator and produces only those, ignoring the variety of the real data.
• Oscillation: the two networks circle around each other instead of converging.
• Vanishing gradients: if the discriminator becomes too good too fast, its verdicts saturate and give the generator almost no signal to learn from.

Because there is no single loss that falls steadily, judging progress is hard too. The usual measure, the Fréchet Inception Distance (FID) from 2017, compares statistics of real and generated images inside a trained image classifier: lower is better.

GANs in the world

The same power enables harm. Deepfakes, a word coined in 2017, put real people's faces and voices into fake videos, and have been used for fraud, harassment and misinformation. Level 15 covers detection, watermarking and the law; Lesson 12.2 shows the newer method that has largely replaced GANs for image generation.

Hands-on: an autoencoder and a GAN

A two-into-one autoencoder

Forty points lie close to the line y = 2x. The encoder squeezes each point's two numbers into one, and the decoder rebuilds two numbers from that one. To rebuild well, the network has to discover the line.

import random

random.seed(1)
data = []
for _ in range(40):
    t = random.uniform(-1, 1)
    data.append((t + random.gauss(0, 0.05), 2 * t + random.gauss(0, 0.05)))

enc = [0.3, -0.2]          # encoder: z = enc . x       (2 numbers squeezed into 1)
dec = [0.1, 0.4]           # decoder: x' = dec * z      (1 number expanded back into 2)
lr = 0.05
for epoch in range(301):
    total = 0.0
    for x in data:
        z = enc[0] * x[0] + enc[1] * x[1]
        rec = [dec[0] * z, dec[1] * z]
        err = [rec[0] - x[0], rec[1] - x[1]]
        total += err[0] ** 2 + err[1] ** 2
        g_dec = [2 * err[0] * z, 2 * err[1] * z]
        g_z = 2 * (err[0] * dec[0] + err[1] * dec[1])
        g_enc = [g_z * x[0], g_z * x[1]]
        dec = [d - lr * g for d, g in zip(dec, g_dec)]
        enc = [e - lr * g for e, g in zip(enc, g_enc)]
    if epoch % 60 == 0:
        print(f"epoch {epoch:3}: reconstruction error {total / len(data):.4f}")

print("decoder direction:", [round(d / dec[0], 2) for d in dec], "(the line y = 2x is [1, 2])")
for x in sorted(data)[:3] + sorted(data)[-3:]:
    z = enc[0] * x[0] + enc[1] * x[1]
    print(f"point ({x[0]:5.2f}, {x[1]:5.2f}) -> latent {z:5.2f} -> rebuilt ({dec[0] * z:5.2f}, {dec[1] * z:5.2f})")

A GAN in one dimension

The real data are numbers with mean 4 and spread 1.25. The generator starts by producing numbers with mean 0 and spread 1, and only ever learns from the discriminator's verdicts.

import math, random

random.seed(2)
def sigmoid(v):
    return 1 / (1 + math.exp(-max(-30.0, min(30.0, v))))

real_mean, real_sd = 4.0, 1.25          # the "real data" the generator never sees directly
a, b = 1.0, 0.0                         # generator: x = a * noise + b
w1, w2, c = 0.0, 0.0, 0.0               # discriminator: D(x) = sigmoid(w1 u + w2 u^2 + c), u = x / 4

def D(x):
    u = x / 4
    return sigmoid(w1 * u + w2 * u * u + c), u

for step in range(1501):
    # 1. the discriminator learns to say 1 for real examples and 0 for fakes
    gw1 = gw2 = gc = 0.0
    for _ in range(8):
        for x, label in ((random.gauss(real_mean, real_sd), 1), (a * random.gauss(0, 1) + b, 0)):
            d, u = D(x)
            gw1 += (d - label) * u
            gw2 += (d - label) * u * u
            gc += d - label
    w1 -= 1.0 * gw1 / 8
    w2 -= 1.0 * gw2 / 8
    c -= 1.0 * gc / 8
    # 2. the generator learns to make the discriminator say 1
    ga = gb = 0.0
    for _ in range(8):
        z = random.gauss(0, 1)
        d, u = D(a * z + b)
        g_x = -(1 - d) * (w1 + 2 * w2 * u) / 4      # how -log D changes as the fake moves
        ga += g_x * z
        gb += g_x
    a -= 0.05 * ga / 8
    b -= 0.05 * gb / 8
    if step % 250 == 0:
        print(f"step {step:4}: fakes have mean {b:5.2f} and spread {abs(a):4.2f};  D(4) = {D(4)[0]:.2f}")
print(f"real data:  mean {real_mean:5.2f} and spread {real_sd:4.2f}")

Watch the spread overshoot and swing back: the two players chase each other around the answer, and D(4) hovers near 0.5 once the fakes are convincing.

Try this

1. In the autoencoder, change the data to lie along y = −x. Which direction does the decoder find?
2. In the GAN, change the generator's learning rate from 0.05 to 0.5. What happens to the spread?

Level check

• Distinguish discriminative and generative models, with examples
• Describe an autoencoder, its bottleneck and its reconstruction loss
• Give three uses of autoencoders
• Explain what a latent space is and what interpolating in it does
• Explain how a VAE makes its latent space suitable for sampling
• Describe how a GAN's generator and discriminator are trained against each other
• Explain mode collapse and why GAN training is unstable
• Discuss the benefits and harms of generated images

Key terms you must be able to define

Generative model

A model that learns what data looks like, so it can produce new examples.

Discriminative model

A model that learns to predict a label from an input.

Autoencoder

An encoder and a decoder trained to rebuild their input through a small bottleneck.

Latent code

The small vector an encoder produces; the compressed description of one input.

Latent space

The space of all latent codes, in which similar inputs lie close together.

Reconstruction error

How different an autoencoder's output is from its input.

Variational autoencoder (VAE)

An autoencoder whose codes are sampled from learned clouds pulled towards a standard bell curve, so new data can be sampled.

Generative adversarial network (GAN)

A generator and a discriminator trained against each other.

Generator

The network in a GAN that turns random noise into fake data.

Discriminator

The network in a GAN that learns to tell real data from fakes.

Mode collapse

When a generator produces only a few kinds of output instead of the data's full variety.

Deepfake

Realistic generated media that puts a real person's face or voice into something they never did.

Flashcards

Go further: notes for future AI engineers

The optimal discriminator

For a fixed generator, the best possible discriminator can be written down exactly. If p_data(x) is how likely x is under the real data and p_g(x) under the generator, then

D*(x) = p_data(x) ÷ (p_data(x) + p_g(x))

Where the generator makes too many of something, D* is below 0.5; where it makes too few, above 0.5. Training the generator against this D* turns out to minimise the Jensen–Shannon divergence, a measure of the difference between the two distributions, which is zero exactly when they are equal, and then D*(x) = 0.5 everywhere.

Wasserstein GANs

When the real and fake distributions barely overlap, the Jensen–Shannon divergence stops changing and the generator's gradients vanish. Wasserstein GANs (2017) instead measure the "earth mover's distance": how much probability has to be moved, and how far, to turn one distribution into the other. It keeps shrinking smoothly as the fakes improve, which made GAN training far more stable.

Misconceptions to avoid