principles.fyi · the brain · concept

residual stream

A shared running notepad every layer reads from and adds its notes back onto, instead of erasing it.

x = x + layer(x)

The residual stream is a vector for each token that travels straight through the whole network, keeping the same width (d_model) at every layer. Instead of replacing it, each layer reads the current value, computes a small update, and adds that update back (x = x + layer(x)) — so contributions accumulate rather than overwriting the slate. Because every layer only adds, information written early stays reachable near the top (later layers can still damp or cancel it, but nothing is wiped by default), and gradients flow back cleanly during training, so very deep stacks still learn. Think of it as a highway each layer merges its contribution onto, rather than a relay where each step rebuilds from scratch. (Layers also normalize their input along the way, but the running stream itself is what carries information forward.)