The modern gated FFN

Frontier models add a second lane that multiplies the signal: a learned soft dimmer instead of a hard on/off.

Classic FFN — one lane

output0.00

Gated / SwiGLU — two lanes

output0.00
-0.80

Slide x below zero: the classic lane slams to 0 the instant ReLU shuts off. The gated lane keeps a faint signal — its gate scales the content by a learned soft amount, fading smoothly instead of snapping. Same widen→think→shrink spirit, gentler hands.