principles.fyi · the brain · concept

cross-entropy loss

A score for surprise: the truer your guess, the smaller it gets.

loss = -log(p_true)

When the model predicts the next word, it spreads 100% of its belief across all possible words, like dividing a coin jar. Cross-entropy loss looks only at the slice it gave the word that actually came next, and asks: how surprised were you? If it bet almost everything on the right word, the surprise is tiny; if it gave the right word only a sliver, the surprise is huge. Training nudges the model to keep raising the slice on true words, so this surprise number shrinks over time.