Agents and Images

How language models use tools, and how transformers process and generate images. Follow tool calls, observations, image patches, and denoising steps.

Contents

1. It can only ever write text — A language model cannot search, click, or check anything. Give its words a reader that acts on them, and one request becomes a loop.
2. Two ways to be wrong — An agent can use observations to check its assumptions and revise its next action. Follow a loop that alternates between deciding what to do and examining the result.
3. A tool is a sentence about a tool — The model cannot see your code. It picks by reading descriptions — so the description is not documentation, it is the whole interface.
4. Tools it never trained on — Weights freeze on a date; APIs do not. So stop putting tools inside the model and hand it the list at the moment of use.
5. Forgetting is not optional — Every observation lands back in a prompt with a hard ceiling. Once you accept that something must be dropped, memory and planning are both just the question of what.
6. Comparing reasoning paths — Keep and compare alternative paths, and distinguish explicit search from a single sequence of reasoning.
7. Where it breaks — The long-horizon wall is arithmetic, not a model flaw. And because actions arrive as text, anything that can write text is holding the same handle you are.
8. Grading something that does things — Code can be run, which makes grading honest. Then two correct solutions look nothing alike, one number hides two questions, and passing tests turns out not to mean fixed.
9. An image is a sequence too — A transformer takes a sequence of vectors and lets every position see every other. Nothing in that says “words” — so the only question an image poses is what one position should be.
10. One space for words and pictures — Nobody could label four hundred million images. But the captions were already attached — and one batch of real pairs generates all the wrong pairs you need.
11. Drawing by subtracting — A diffusion model learns to denoise images at different noise levels. Repeated denoising steps turn a random starting point into a generated image.
12. Telling it what to draw — A denoiser needs information about the noise level and the image to generate. Compare ways to supply a class label or text prompt to a transformer.

Sources and further reading are in the bibliography.