An image cut into patches of varying size, showing the trade between how much one token holds and how many tokens attention must compare.

Nothing about a transformer says “words”

It takes a sequence of vectors and lets every position look at every other. Hand it an image and only one question matters: what is one position?

Patch size changes the input projection. An RGB patch has 3p² values, so its projection uses 3p² × d weights, excluding bias. Halving p gives four times as many tokens and sixteen times as many attention pairs. Transformer block weights can stay the same size; the whole trained model cannot generally stay unchanged.