principles.fyi · the brain · concept

logits

A score for every possible next word, before turning them into probabilities.

softmax(x)_i = e^(x_i) / sum_j e^(x_j)

Logits are the model's raw, unnormalized scores — one number per word in the vocabulary — saying how much it favors each as the next word. They come straight from the final layer and can be any real number: a higher logit means "more likely" relative to the others, and only the differences between them matter (adding the same amount to every logit changes nothing). Softmax then squashes the whole list into probabilities that add up to 1, via softmax(x)_i = e^(x_i) / sum_j e^(x_j), so the highest logit becomes the most probable word.