principles.fyi · the brain · concept

nonlinearity

A bend in the math that lets stacked layers learn more than a straight line.

ReLU(x) = max(0, x)

A nonlinearity is a simple function that bends its input instead of just scaling it—ReLU, for example, keeps positive numbers unchanged and sets negative ones to zero. It matters because stacking matrix multiplies with nothing in between is mathematically the same as one big matrix multiply, so extra depth would buy you nothing. Inserting a bend between layers lets the network stack many transformations and carve out curved, complex patterns. In a transformer, the main activation-function bend (ReLU or GELU) lives inside the feed-forward block after attention—though it's not the only nonlinearity around, since the softmax in attention and layer norm are nonlinear too. Together these bends give each layer real expressive power instead of collapsing into a single linear map.