principles.fyi · the brain · concept

feed-forward network

A small per-word network that expands, filters, then re-compresses its meaning.

FFN(x) = W2 * nonlinear(W1 * x + b1) + b2

It's a two-layer transformation applied to each token on its own: W1 projects the vector into a much wider space, a nonlinearity zeroes out or bends parts of it, then W2 shrinks it back to size. The wide middle gives the network room to recognize and react to many distinct patterns at once — like having thousands of tiny "if this feature is present, add that" rules. While attention mixes information between tokens, the feed-forward network does the per-token thinking, and its weights hold much of the factual and stylistic knowledge the model has learned.