The feed-forward network: a 4-dimensional vector is projected up to 8 dimensions by W_up, passed through ReLU which zeroes negatives, then projected back down to 4 dimensions by W_down. Click stages to see each matrix multiplication.

What a feed-forward layer does

Widen the vector, keep the positives, fold it back down.

Width profile of the feed-forward network Four stacks of cells: 4, then 8, then 8, then 4 — showing the vector widening then narrowing. Matrix multiplication detail for the active stage A vector multiplied by a weight matrix to produce the next vector.