Drawing by subtracting

A diffusion model can generate an image through a sequence of denoising updates. The useful training trick is that we can make noisy examples ourselves.

Start with a training image, choose a noise level, and combine the image with sampled noise. In one common version of diffusion training, the network learns to predict that noise from the noisy input and its noise level. The target is known because we sampled it.

During generation, the clean image is unknown. A sampling algorithm starts from noise and repeatedly uses the trained network’s predictions to update the current sample.

What the network predicts

In the noise-prediction version, the network outputs a noise estimate. The sampler combines it with the current sample and a noise schedule to calculate the next update. This is more specific than simply subtracting the displayed middle picture. Other diffusion formulations can predict a clean sample or another related quantity.

The same trained network can be used at multiple steps. It also receives the noise level, because an almost-random input and a nearly-clean input call for different predictions. More sampling steps spend more computation; the best number depends on the model and sampling method.

The illustration above cannot demonstrate which visual structures a trained model learns first. Every frame is constructed from the same known drawing.

Do it somewhere smaller

Denoising a full-resolution image can be expensive. Latent diffusion reduces that cost by working with a compressed representation.

An autoencoder might compress a 256×256×3 image to a 32×32×4 latent array. The denoising loop operates on that array, and a decoder reconstructs an image at the end. These dimensions are an example, not a requirement of every diffusion model.

What predicts the noise?

We have been calling it “the model.” It has to take a grid and return a grid of the same shape. Convolutional U-Nets did this for years.

But the last chapter already turned a grid into a sequence. So: cut the noisy latent into patches, project each to a vector, add positions, run a transformer, and reshape the output back into a grid of noise. A diffusion transformer — a vision transformer, fed a noisy latent, predicting noise instead of a class.

As in the image-patch example, smaller patches give more tokens at a fixed resolution: halving the patch size gives four times as many tokens and sixteen times as many full-attention pairs. The transformer block widths can stay fixed, but patch-dependent input and output projections change shape. The DiT paper compared trained models with different patch sizes; this is not generally a setting to change on an existing model without adaptation.

Go deeper: how do you score a picture?

Human judgement does not scale, and there is no reference image to compare against — the whole point is that this one is new. There is also a failure that a per-image score would miss entirely: a model that produces one excellent picture every time is useless, and every individual output looks fine.

So compare distributions. Run a pretrained classifier over a pile of real images and a pile of generated ones, take the activations from a late layer, and measure the distance between the two clouds of numbers — the Fréchet Inception Distance.

It catches both faults at once: generated images that do not look like real ones move the cloud, and generated images that are all alike shrink it. Lower is better. It is a proxy, it can be gamed, and it is what the field runs on.

So far it can generate an image. It cannot yet generate the one you asked for.

Sources · 4

Full bibliography →