principles.fyi · the brain · concept

exponential (e^x)

Growth that speeds up as it goes: a tiny step makes a giant leap.

e^x: e^0 = 1, e^1 ≈ 2.7, e^2 ≈ 7.4

e^x is a curve where the bigger the number gets, the faster it grows. For negative x it hugs the floor, almost nothing; it passes through 1 when x is zero; then for positive x it rockets upward. Softmax feeds scores through e^x, so a small lead in score becomes a big lead in the final probability. It is the exact mirror of the logarithm, which slowly squashes big numbers back down.