principles.fyi · the brain · concept

ReLU

A switch that lets positive numbers through and turns negatives to zero.

ReLU(x) = max(0, x)

ReLU ("Rectified Linear Unit") is a tiny rule applied to each number: keep it if it's positive, replace it with 0 if it's zero or negative — that's all of ReLU(x) = max(0, x). This bend is what lets a network learn curved, complicated patterns instead of only straight-line ones; without a nonlinear step like this, stacked layers would collapse into one plain linear map. Pile up many of these bends and the network can approximate almost any shape. Inside a transformer it's a classic choice for the feed-forward layer that follows each attention step, where it decides which signals each neuron passes onward and which it silences (though many modern transformers swap it for cousins like GELU).