Transformers, ELI5 · Part 7 / 10

From scores to a word — and the loop

Softmax makes probabilities, temperature and top-k/top-p shape the pick, then the chosen word is appended and the whole model runs again.

One logit per word. Two steps left: make them probabilities, pick one. Then close the loop from Chapter 1.

Softmax: scores → probabilities

Softmax turns the logits into probabilities (summing to 1) and blows the leader up — a slightly higher score becomes a much higher one. (It does this by exponentiating with e.)

Temperature: bold or safe?

Divide the logits by a temperature first:

• Low (0.3): spiky — the top word dominates. Safe, repetitive.
• High (1.5): flat — longshots get a chance. Varied, surprising.

Try it: drag to 0.2 (one bar wins), then 1.8 (flat). Sample a few times — different words come out.

# logits: the model's raw scores for the next token  # logits: (vocab,)
import torch

temperature = 0.8  # <1 = sharper/safer picks, >1 = flatter/more random, 1 = unchanged

scaled = logits / temperature           # squash or spread the scores  # (vocab,)
probs = torch.softmax(scaled, dim=-1)   # turn scores into probabilities that sum to 1  # (vocab,)

# sample: roll a weighted die, so likely tokens win more often (but not always)
next_token = torch.multinomial(probs, num_samples=1)  # (1,)

# greedy alternative: always grab the single most likely token (no randomness)
next_token_greedy = torch.argmax(probs, dim=-1)  # ()

Trimming the field: top-k & top-p

Sampling from every word risks the bad long tail (and always taking the top — greedy — gets repetitive; you can also fight loops directly with a repetition penalty). So cut it first:

• Top-k — keep the k likeliest words.
• Top-p — keep the fewest words that add up to p. Adapts: few when confident, many when unsure.
• Min-p — newer: keep words at least a fraction (say 5%) as likely as the top word. The bar rises with the model’s confidence.

Try it: in Top-k, drag k. In Top-p, drag p and watch the number of kept words change on its own.

Close the loop

Pick a word — say on. Append it: “the cat sat on”. Run the whole network again for the next word. And again.

A model doesn’t write sentences. It writes one word, glues it on, and re-reads everything.

import torch

# `model` maps a sequence of token ids -> logits for every position.
# logits[..., t, :] = scores over the whole vocab for "what comes after position t".

tokens = torch.tensor([[1, 14, 27]])   # our prompt so far: (batch=1, seq=3)

for _ in range(5):                      # generate 5 new tokens, one at a time
    logits = model(tokens)              # run model on ALL tokens so far: (1, seq, vocab)
    last_logits = logits[:, -1, :]      # we only care about the LAST position: (1, vocab)
    probs = torch.softmax(last_logits, dim=-1)   # turn scores into probabilities
    next_token = torch.multinomial(probs, num_samples=1)  # sample one token: (1, 1)
    tokens = torch.cat([tokens, next_token], dim=1)       # append it -> seq grows by 1

# after the loop, `tokens` holds prompt + 5 freshly generated tokens
print(tokens)

That’s the core machine

text → vectors → attention + FFN ×96 → last vector → logits → softmax → sample → append → repeat

Nearly all of it, as promised, is matrix multiplication.

Three things left: where the weights came from (training), how models get so big (scale), and whether we can peek at what they learned.