principles.fyi · the brain · concept

sampling

Rolling a weighted die to pick the next word.

p_i = e^(logit_i) / sum_j e^(logit_j), then sample from p

The model scores every possible next token with a raw number (a "logit"). Those raw scores aren't probabilities yet — they can be negative and don't add up to 100% — so the model first squashes them through a step called softmax that turns the whole list into clean percentages that sum to 1. Then, instead of just grabbing the single highest one, it rolls a weighted die over those percentages: a token that ends up at 30% gets picked about 30% of the time. That randomness is why the same prompt can give different answers, and why the text feels lively instead of robotic — always taking the top choice tends to produce flat, repetitive writing, while letting lower-ranked words occasionally win adds variety and surprise. The "temperature" knob acts before the softmax, stretching or squeezing the raw scores to tilt the die toward safe-and-likely (low temperature) or wild-and-creative (high temperature).