Telling it what to draw

The denoiser has a problem it cannot solve internally. It needs to know two things that are not patches: how noisy its input currently is, and what the picture is meant to be of.

A transformer block takes a sequence and returns a sequence. There is no third input. So this is a plumbing question with several honest answers, and it is worth seeing them lose.

The winner does not join the conversation

The first two options treat the condition as content — something the block should read. Append it as extra tokens, or add a layer that reaches over to it.

The third stops trying. Every block already normalises its activations, then rescales and re-shifts them. Those scale and shift numbers are usually learned constants. So compute them from the condition instead.

Now the condition never enters the sequence. It sets the dials — how strongly this layer scales, how far it shifts, how much of each sublayer’s output survives. It stops being something the block reads and becomes something that changes how the block behaves. Adaptive layer normalisation: negligible cost, best pictures.

There is a lesson in that beyond image generation. Two mechanisms were available — inform the computation or modulate the computation — and for a signal that applies globally and uniformly, modulation was both cheaper and better.

A prompt was never a label

All three assume the condition is one vector. That is fine for class 207: golden retriever.

Now try “a red bicycle leaning against a green door, late afternoon”. Squeeze that into a single vector and you have thrown away everything that made it a sentence — which object is red, which is green, what is leaning on what. The information you most needed is exactly the information a single vector cannot hold.

So stop squeezing. Keep the text as a sequence, the way a text transformer always did.

Two streams, then: text tokens and noisy patch tokens. They want different weights — words and image patches are different enough that sharing parameters helps neither. But they must meet, or the picture cannot be about the words.

They meet in one attention operation. Both streams keep their own normalisation and their own feed-forward layers, and in the middle every text token and every patch attend over each other together — joint attention. Two towers, one conversation. That is MM-DiT, and it is what current text-to-image models are.

The timestep still arrives by adaptive layer norm, because how noisy is this really is one number that applies everywhere. Each signal enters by the route that fits its shape.

Go deeper: turning the prompt up

There is one more trick, and it is almost embarrassing. Run the denoiser twice — once with the prompt, once without — and you get two predicted noises. Their difference is the direction the prompt is pushing in.

Then overshoot it. Take the unconditioned prediction and step further along that difference than the model actually asked for. Prompts get followed harder, images get more vivid, and pushed too far, images get lurid and lose variety.

That is classifier-free guidance, it costs double compute per step, and it is the knob behind a “prompt strength” slider.

What the whole module was about

Two halves, and one idea underneath them.

A language model does one thing: read a sequence of vectors, write a sequence of vectors. Part one changed what happens to its output — let a harness read some of it as commands, and a text predictor becomes something that searches, queries, edits and browses. Everything after that was consequence: tools are descriptions because descriptions are all it can see; forgetting is mandatory because the loop refills a fixed prompt; injection works because actions arrive as text and so does everything else.

Part two changed what goes in. A patch is a vector, so an image is a sequence. Captions already sat under pictures, so one space could hold both. And a picture can be produced by a transformer too — not by drawing it, but by repeatedly naming the noise.

Then the halves close on each other. The agent from chapter one, given the eyes from chapter ten, reads a rendered page — and hallucinates a button, because it is still only ever predicting what comes next.

Sources · 3

Full bibliography →