Diffusion Models and Multimodal AI

Destroy, then learn to repair

A diffusion model learns to generate by learning to undo damage.

• The forward process takes a real image and adds a little random noise, again and again, for hundreds of steps, until nothing is left but noise. This part needs no learning at all.
• The reverse process is learned: a network is trained to remove a little noise at a time. Run it from pure noise and an image appears.

After t steps, the noisy image is a mix of the original and fresh noise:

With the 2020 schedule of 1,000 steps, after step 250 the mix is 0.72 × image + 0.69 × noise; after step 500, 0.28 × image + 0.96 × noise; and by step 1,000, almost pure noise.

Training the denoiser

Training is a simple loop:

1. Take a training image and pick a random step t.
2. Make random noise, and mix it with the image as the forward process would at step t.
3. Ask the network: given this noisy image and the number t, what noise was added?
4. The loss is the squared difference between the real noise and the predicted noise.

The network is usually a U-Net, a convolutional network that shrinks the image to find its overall layout and then grows it back, with skip connections carrying fine detail across; newer models use transformers instead. Because the network only ever has to remove a little noise, every training example is a small, well-defined task.

Generating: noise to picture

To generate, start with pure random noise and repeat: predict the noise, remove some of it, and add back a little less noise than before. Over 20 to 1,000 steps an image condenses out of the static.

Early steps decide the overall layout and colours; later steps add edges and fine detail. Because every run starts from different random noise, every run gives a different image.

The lab below does exactly this to points on a plane: drag the steps, pick a shape, and watch noise flow into it.

Steering with words

To draw "a red fox in the snow", the denoiser needs to know what was asked. The prompt is turned into embeddings by a text encoder, and the denoiser reads them through cross-attention (Level 10): every patch of the image can attend to every word.

Classifier-free guidance (2021) makes the image follow the prompt more closely. At each step the network predicts the noise twice, once with the prompt and once without, and the step is pushed further in the direction the prompt made:

A higher guidance scale follows the prompt more literally but gives less varied, sometimes over-saturated images.

Latent diffusion

A 512 × 512 colour image holds 786,432 numbers, and denoising all of them for every step is slow. Latent diffusion (2021), the method behind Stable Diffusion (2022), first trains an autoencoder (Lesson 12.1) that squeezes an image into a 64 × 64 × 4 latent: 16,384 numbers, 48 times fewer. Diffusion then happens entirely in that latent space, and the decoder turns the final latent into the picture.

CLIP: one space for pictures and words

How does a text encoder know which words go with which pictures? CLIP (2021) was trained on 400 million images with their captions from the web. It has two encoders, one for images and one for text, trained with contrastive learning:

In every batch, the true image–caption pairs, on the diagonal, are pushed to have high cosine similarity, and every mismatched pair is pushed low. The result is one shared space where a photo of a dog lands near the words "a photo of a dog".

That enables zero-shot classification: to sort images into new categories, embed captions such as "a photo of a fox" and "a photo of a hare", and pick the caption closest to the image, with no extra training.

Models that see, hear and speak

A multimodal model works with more than one kind of data: text, images, audio or video.

The common trick is to turn every kind of data into tokens or embeddings: image patches (Level 10's Vision Transformer), slices of audio, frames of video. Once everything is a sequence of vectors, one transformer can attend across all of them.

Real or generated?

Generated images and voices are now good enough to fool people, and detectors that guess from the pixels alone are unreliable. Two other approaches help:

• Watermarking: the generator hides a signal in its output, invisible to people but detectable by software, as Google DeepMind's SynthID (2023) does for images, audio and text.
• Provenance: standards such as C2PA content credentials attach signed records of how a file was made and edited, so a photo can show that it came from a real camera.

Generative AI also raises questions of consent and copyright: models are trained on artists' work, photographers' photos and people's faces, often without asking. Level 15 returns to these questions.

Hands-on: diffusion from scratch

How much image is left?

import math

T = 1000
betas = [0.0001 + (0.02 - 0.0001) * t / (T - 1) for t in range(T)]   # a little more noise each step
alpha_bar, keep = [], 1.0
for b in betas:
    keep *= 1 - b
    alpha_bar.append(keep)

for t in [1, 50, 100, 250, 500, 750, 1000]:
    signal = math.sqrt(alpha_bar[t - 1])
    noise = math.sqrt(1 - alpha_bar[t - 1])
    print(f"after step {t:4}: noisy image = {signal:.3f} x image + {noise:.3f} x noise")

Noise into a ring

The training data are 60 points on a ring. With so little data, the best possible denoiser can be written exactly: a softmax-weighted average of the data points, weighted by how well each explains the noisy point. That is the same softmax you used for attention. Each generated point starts as pure random noise.

import math, random

random.seed(0)
data = [(math.cos(2 * math.pi * k / 60), math.sin(2 * math.pi * k / 60)) for k in range(60)]   # a ring

T = 40
def alpha_bar(t):                       # how much of the clean point survives after t steps
    return math.cos((t / T + 0.008) / 1.008 * math.pi / 2) ** 2

def denoise(x, t):
    # The best possible guess of the clean point: a softmax-weighted average of the data
    a, s = math.sqrt(alpha_bar(t)), math.sqrt(1 - alpha_bar(t))
    scores = [-((x[0] - a * d[0]) ** 2 + (x[1] - a * d[1]) ** 2) / (2 * s * s) for d in data]
    m = max(scores)
    w = [math.exp(v - m) for v in scores]
    total = sum(w)
    return (sum(wi * d[0] for wi, d in zip(w, data)) / total,
            sum(wi * d[1] for wi, d in zip(w, data)) / total)

def generate():
    x = (random.gauss(0, 1), random.gauss(0, 1))          # start from pure noise
    for t in range(T, 0, -1):
        a, s = math.sqrt(alpha_bar(t)), math.sqrt(1 - alpha_bar(t))
        x0 = denoise(x, t)
        eps = ((x[0] - a * x0[0]) / s, (x[1] - a * x0[1]) / s)         # the noise it thinks is there
        a2, s2 = math.sqrt(alpha_bar(t - 1)), math.sqrt(1 - alpha_bar(t - 1))
        x = (a2 * x0[0] + s2 * eps[0], a2 * x0[1] + s2 * eps[1])        # remove a little of it
    return x

def plot(points):
    grid = [[" "] * 41 for _ in range(19)]
    for x, y in points:
        col, row = round(20 + x * 13), round(9 - y * 6)
        if 0 <= row < 19 and 0 <= col < 41:
            grid[row][col] = "o"
    for r in grid:
        print("|" + "".join(r) + "|")

samples = [generate() for _ in range(80)]
plot(samples)
radius = sum(math.hypot(x, y) for x, y in samples) / len(samples)
print(f"average distance from the centre: {radius:.3f} (the ring has radius 1)")

Zero-shot matching, CLIP style

The embeddings below are made up, four numbers each, to show the idea; CLIP's have 512.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

photo = [0.9, 0.1, 0.8, 0.0]            # pretend image embedding: a dog on a beach
captions = {
    "a photo of a dog": [0.8, 0.0, 0.3, 0.1],
    "a photo of a cat": [0.1, 0.9, 0.2, 0.0],
    "a photo of a beach": [0.3, 0.0, 0.9, 0.1],
    "a photo of a dog on a beach": [0.7, 0.0, 0.7, 0.05],
    "a diagram of a car engine": [0.0, 0.1, 0.0, 0.9],
}
for text, vec in sorted(captions.items(), key=lambda kv: -cosine(photo, kv[1])):
    print(f"{cosine(photo, vec):.3f}  {text}")

Try this

1. In the ring program, change T to 5 and then to 200. How does the picture change?
2. Replace the ring with two clusters, 30 points near (−1, 0) and 30 near (1, 0). What do the generated points do?

Level check

• Describe the forward and reverse processes of a diffusion model
• Explain how the denoiser is trained to predict noise
• Explain how text steers generation through cross-attention and classifier-free guidance
• Explain why latent diffusion is faster than diffusion on pixels
• Describe how CLIP is trained with contrastive learning and how zero-shot classification works
• Give examples of multimodal models and how they turn every kind of data into vectors
• Explain watermarking and provenance for generated media

Key terms you must be able to define

Diffusion model

A generative model that creates data by learning to remove noise, step by step.

Forward process

Adding noise to data over many steps until only noise remains; it involves no learning.

Reverse process

The learned, step-by-step removal of noise that turns pure noise into data.

Noise schedule

How much noise is added at each step of the forward process.

U-Net

A convolutional network that shrinks and then regrows an image, with skip connections, used as a denoiser.

Classifier-free guidance

Pushing each denoising step further in the direction the prompt makes, to follow the prompt more closely.

Latent diffusion

Running diffusion on an autoencoder's small latent version of an image instead of its pixels.

CLIP

A pair of image and text encoders trained so that matching images and captions have similar embeddings.

Contrastive learning

Training that pulls matching pairs together and pushes mismatched pairs apart.

Zero-shot classification

Classifying into categories never trained on, by comparing with embeddings of their descriptions.

Multimodal model

A model that works with more than one kind of data, such as text, images and audio.

Watermark

A hidden signal in generated content that software can detect.

Provenance

A signed record of where a piece of media came from and how it was edited.

Flashcards

Go further: notes for future AI engineers

Memorising versus generalising

The ring program's denoiser is perfect for its 60 training points, and that is a problem: every generated point lands exactly on a training point, so it can only ever reproduce its data. Real diffusion models generalise because their networks cannot store billions of images exactly and are forced to learn smooth rules instead. Researchers have still found that large models occasionally reproduce near-copies of images that appeared many times in their training data, which matters for privacy and copyright.

Fewer steps

A thousand network evaluations per image is slow. Deterministic samplers such as DDIM (2020), the update the ring program uses, give good images in 20 to 50 steps, and distillation trains a student to jump through several steps at once, down to one to four steps. Flow matching, used by several recent image models, learns straighter paths from noise to data that need fewer steps to follow.

Misconceptions to avoid