Scale and computation

Scaling, cached attention, and low-rank updates

Increasing model size can improve prediction, but also increases the resources required. This chapter looks at measured scaling trends, what they tell us about capabilities, and ways to reduce computation or training costs.

Predictably better

Scaling laws are empirical relationships between language-model loss, often measured on held-out text, and resources such as parameters, data, and compute. Over studied ranges, these relationships can often be approximated with power laws. They help estimate tradeoffs; they do not guarantee that a larger model performs better on every task.

Go deeper: it's a budget you split (Chinchilla)

Params aren’t the only dial. For a fixed compute budget you split it between model size and training data — and there’s a sweet spot. That’s compute-optimal scaling.

Did few-shot learning appear at one model size?

Few-shot prompting supplies a few examples in the input, then asks for another answer. For example, show two reviews with their sentiment labels, then provide an unlabeled review. The examples change the model’s input; they do not update its weights. Those weights were learned during earlier training.

The GPT-3 study tested eight model sizes, from 125 million to 175 billion parameters. Larger models often benefited more from examples, but most tested tasks showed relatively smooth improvement. The paper did not establish one parameter count at which few-shot learning universally switches on. Brown et al., 2020

Some tasks do show a sharper-looking change: smaller models score near chance, then larger ones do substantially better. Researchers called these emergent abilities. That observation concerns a particular task, model family, and scoring method. Training data and training choices also matter; parameter count is not a universal boundary. Wei et al., 2022

The scoring method can make a gradual improvement look sudden. Imagine four hypothetical models choosing between two labels. They always pick the label with the higher probability:

	Model A	Model B	Model C	Model D
Probability assigned to the correct label	40%	49%	51%	60%
Chose the correct label?	No	No	Yes	Yes

The probabilities improve gradually. A right-or-wrong score jumps when the correct label becomes more likely than the other one. These are invented numbers for one question, not measured model results. Schaeffer and colleagues demonstrated that metric choices can produce apparent emergence in studied benchmarks. That cautions against reading every sharp graph as a sudden internal transformation; it does not prove that every capability change is an illusion. Schaeffer et al., 2023

What does that say about intelligence?

Using examples to handle new cases is an ability worth understanding. The next question is how far it carries: does the model still succeed when the wording changes, labels are unfamiliar, or the problem needs a different combination of steps? Those tests help distinguish a reusable ability from success on a familiar setup.

A parameter count tells us the size of a model. A benchmark tells us how it performed under particular conditions. Neither, by itself, settles what we mean by intelligence or how broadly the model can reason. Looking inside the computation and testing its behavior address different parts of that question.

A larger parameter count is one way to change a model. Recurrence across depth explores a different question: what happens when the same learned block performs more internal updates?

Don’t redo the past: the KV cache

The KV cache stores keys and values already computed for earlier tokens. For the next token, the model computes the new representations and attends to the stored history. It still reads earlier keys and values; it avoids recomputing them.

Count attention pairs, not every operation

With t visible tokens, a full causal attention matrix has t(t + 1)/2 allowed query–key pairs. Recomputing it costs quadratic work in t for those pairs. A cached step needs only t pairs for the new query. Generating a sequence of increasing length therefore has different totals: cubic for repeated full-prefix attention versus quadratic for cached attention, ignoring a fixed prompt length and holding model width and layer count fixed.

The cache stores key and value vectors at each layer. Projection, feed-forward, memory-transfer, and initial prompt-processing costs also matter, so this pair count is not a complete latency forecast.

Full attention over n tokens involves roughly n² token pairs. The supported input length is the context window. Applications must decide how to handle longer conversations, for example by truncating or summarizing earlier text.

Fine-tune cheaply: LoRA

LoRA freezes a pretrained weight matrix and learns an update expressed as the product of two smaller matrices. For a 1,024 × 1,024 matrix, a rank-8 update uses 1,024 × 8 + 8 × 1,024 = 16,384 trainable parameters, compared with 1,048,576 in the full matrix. The choice of rank limits the updates it can represent. See the LoRA paper.

Go deeper: two more 'big but cheap' tricks

Quantization represents values with fewer bits, reducing storage at the cost of approximation error. Mixture of experts routes each token through a subset of available subnetworks, reducing computation relative to using them all.

Last stop: can we see what the model learned?

Sources · 12

Full bibliography →