Tokens, vectors, and similarity

Embeddings, dot products, and position information

A token ID is a lookup address. Its numerical size says nothing about meaning: token 100 is not twice as meaningful as token 50.

A word becomes a list of numbers

Give every token a learned list of numbers: its embedding. The invented vectors below have four entries.

cat → [0.9, 0.3, -0.2, 0.6]
dog → [0.8, 0.4, -0.1, 0.5]

A trained model learns its embedding table. These two vectors were chosen for illustration; they are not measurements from a particular trained model.

see this in PyTorch

Each token id is just a number that looks up its own learned vector in a big table, turning a row of ids into a stack of meaning-vectors.

import torch
import torch.nn as nn

vocab_size = 128000   # how many distinct tokens exist (the whole "dictionary")
d_model = 64         # how big each token's vector is (its "meaning" in numbers)

# The learned lookup table: one row per token, each row is a d_model vector.
# embedding.weight has shape (vocab_size, d_model) and is trained like any layer.
embedding = nn.Embedding(vocab_size, d_model)

# Our input is just token ids — integers pointing at rows in the table.
ids = torch.tensor([42, 7, 1001])   # ids: (seq,)  here seq = 3

# Look up each id -> grab its row -> stack into one tensor.
vectors = embedding(ids)   # vectors: (seq, d_model) = (3, 64)

# So token id 42 became a 64-number vector, id 7 its own vector, etc.
print(ids.shape)      # torch.Size([3])
print(vectors.shape)  # torch.Size([3, 64])

Go deeper: how does text even become numbers?

The text input is first split into tokens (whole words, or fragments like straw+berry), each a row-number in a fixed dictionary — and that number is what gets looked up.

Go deeper: how do the numbers learn their meaning?

The table usually starts from random values. Training changes those values along with the other parameters to improve prediction. Related uses can lead to related vectors, but a useful semantic arrangement is learned rather than assigned by a human. The animation below is a schematic illustration, not a live training run.

A list of numbers is an arrow

A vector can be drawn as an arrow from the origin. Angle and length describe relationships between vectors, but neither is a complete definition of a token’s meaning. Cosine similarity compares directions; the dot product also depends on both lengths.

An input embedding is the same whenever that token ID is looked up. After attention and feed-forward updates, its contextual representation depends on the surrounding sequence.

Scoring a guess is a dot product

An output score can be a dot product between a contextual vector h and a token’s output vector w. For $h=(2,1)$ and $w=(3,-1)$ ,

h\cdot w=2(3)+1(-1)=5.

The result is one score, not a probability. An output vector need not be the same vector used for the input embedding. Weight tying is the choice to share those parameters.

see this in PyTorch

A matrix multiply (@) takes a grid of numbers shaped (n, d) and a grid shaped (d, k) and produces a new (n, k) grid, where the inner sizes must match and each output is a row dotted with a column.

import torch

# A tensor is just a grid of numbers (like a spreadsheet of values the model learns from).
A = torch.randn(2, 3)   # A: (n, d) = (2 rows, 3 cols) -> 2 tokens, each a 3-number vector
B = torch.randn(3, 4)   # B: (d, k) = (3, 4)            -> turns each 3-vector into a 4-vector

# The shape rule for @ : (n, d) @ (d, k) -> (n, k)
# The inner d's MUST match (3 == 3); they "cancel", leaving the outer (n, k).
C = A @ B               # same as torch.matmul(A, B)
print(C.shape)          # torch.Size([2, 4])  -> 2 tokens, now described by 4 numbers each

# Each output number is a row of A "dotted" with a column of B:
# C[0, 0] = A[0, 0]*B[0, 0] + A[0, 1]*B[1, 0] + A[0, 2]*B[2, 0]  (multiply pairs, then add)
print(torch.allclose(C[0, 0], (A[0] * B[:, 0]).sum()))  # True

Where each word sits

Swapping tokens swaps their rows, so the input matrices already differ. But unrestricted self-attention without position information treats a row permutation consistently; it has no explicit representation of token positions. Additive position embeddings give each position its own vector and add it to the token vector.

Next: how words borrow meaning from their neighbours.

Sources · 6

Full bibliography →