principles.fyi · the brain · concept

perplexity

How surprised a model is by real text it didn't train on — lower means it predicted better.

perplexity = e^(cross-entropy) = exp(average surprise per token) (lower = better)

Perplexity measures how well a model predicts a held-out piece of text it never trained on: at each word it had a probability for what actually came next, and perplexity rolls those up into one 'how surprised was I, on average' number. It's just Book 01's cross-entropy loss run back through an exponential, so a perplexity of 10 roughly means 'at each step the model was about as unsure as if it were picking between 10 equally-likely words.' Lower is better. The catches: it only fairly compares models that chop text into the same tokens (the same tokenizer), and being un-surprised by text is not the same as being correct, helpful, or safe — so it's a useful proxy, not the whole story.