Transformers, ELI5 · Part 9 / 10

Making it big, affordably

Bigger models are predictably better — and a few tricks (the KV cache, LoRA) keep the cost in check.

Bigger models are better. The catch is cost — so there are clever tricks.

Predictably better

Add parameters, data, or compute and the loss falls along a straight line on log-log axes — a power law. These predictable curves are called scaling laws — you can predict the payoff before you spend a cent.

Don’t redo the past: the KV cache

For each new word, the model would re-attend over the whole history every time. The KV cache stores the past instead, so it only computes the new row.

import torch

# `model` maps a sequence of token ids -> logits for every position.
# logits[..., t, :] = scores over the whole vocab for "what comes after position t".

tokens = torch.tensor([[1, 14, 27]])   # our prompt so far: (batch=1, seq=3)

for _ in range(5):                      # generate 5 new tokens, one at a time
    logits = model(tokens)              # run model on ALL tokens so far: (1, seq, vocab)
    last_logits = logits[:, -1, :]      # we only care about the LAST position: (1, vocab)
    probs = torch.softmax(last_logits, dim=-1)   # turn scores into probabilities
    next_token = torch.multinomial(probs, num_samples=1)  # sample one token: (1, 1)
    tokens = torch.cat([tokens, next_token], dim=1)       # append it -> seq grows by 1

# after the loop, `tokens` holds prompt + 5 freshly generated tokens
print(tokens)

Because attention compares every token with every other (~n² work), a model can only hold so many at once — its context window (commonly ~128k, up to ~1M+ tokens). Run past it and the start of the chat falls off.

Fine-tune cheaply: LoRA

Re-training a giant weight matrix means changing millions of numbers. LoRA freezes it and learns a tiny shortcut beside it — a handful of numbers instead of millions.

Last stop: can we see what the model learned?