principles.fyi · the brain · concept

weights

The learned numbers a model tunes during training to store what it knows.

output = weights · input (each output is a weighted sum of the inputs)

Weights are the actual numbers filling the model's matrices — millions or billions of them. Each one controls how strongly one piece of information pushes on another as data flows through; training nudges every weight up or down so the model's guesses get less wrong, and the final settings are what the model "knows." Inside a transformer, weights don't directly pick which words attend to which — that pattern is computed fresh for each input. Instead, the weights build the lenses (the query/key/value projections) that turn each word into the cues used to figure out that attention pattern on the fly, and they reshape each word's meaning at every layer. That's why the same architecture becomes a poet or a coder purely by having different weights.