principles.fyi · the brain · concept

layer norm

It rescales each token's numbers to a common range so training stays steady.

y = (x - mean(x)) / sqrt(var(x) + eps) * gain + bias

Layer norm takes one token's vector of numbers and, using only that vector's own values, shifts and stretches them so they average around zero with a consistent spread (mean ~0, unit variance). The intuition: as activations pass through a deep stack they tend to blow up or shrink, and renormalizing keeps every step in a comfortable range so gradients flow and learning doesn't stall. It runs independently per token (normalizing across that token's feature dimension, not across the batch), and is applied around each attention and feed-forward block — before the block in modern Pre-LN designs, or after the residual add in the original Post-LN design. It also includes two learned per-feature knobs, a gain (scale) and a bias (shift), so the model can re-stretch or re-shift the normalized values if that helps.