principles.fyi · the brain · concept

standard deviation (spread)

How far apart the numbers are: bunched up or spread out.

std = sqrt( average of (each value - mean)^2 )

First find the mean (the balance point). Standard deviation is the typical distance each number sits from that mean: small when they huddle close, large when they wander far. It's one number that captures "how spread out is this pile?". Layer norm subtracts the mean and divides by the spread, so a tight vector and a scattered one come out the same overall size.