principles.fyi · the brain · concept
top-p (nucleus) sampling
Keep just enough likely words to cover p of the total probability, then pick one at random from those.
smallest set S where sum of P(token) for token in S >= p
After the model assigns a probability to every possible next word, top-p sorts them high to low and keeps adding words until their cumulative probability first reaches p (say 0.9) — the smallest group that crosses that line — throwing away the long tail. It then samples the next word from only that "nucleus." The clever part is that the set resizes itself: when the model is confident, a few top words already cover p, so choices stay safe; when it's unsure, the set grows wider, allowing more variety. This keeps writing fluent by avoiding bizarre rare words, while still leaving room for creativity.
Appears in
- From scores to a word — and the loop Transformers, ELI5 · pt 7