Change one idea. See what changes in the calculation.
How much of the past do we read?
Each numbered tile is a token position. Move the query earlier: later positions must become unavailable in every mode.
Solid border: read. Dashed border: not read (outside the local window or not selected). Crossed out: future. A joined tile represents one compressed entry.
16context entries retained
16entries read by this query
What this small model leaves out
In the compression modes, each pair becomes one placeholder for a learned summary vector. We do not compute that vector. The supplied shortlist selects the group containing position 3 and the newest distinct group. A real indexer must find useful groups from data, which costs computation and can miss useful information.
These counts describe one query’s context entries. They exclude index scoring, projections, local branches, vector widths, caching overhead, and all other layers. They are neither total FLOPs nor predicted speedups for DeepSeek.
Store four rules. Run only the selected ones.
Here every expert receives the number x = 2. The router assigns scores; we select the highest scores and normalize them to sum to 1.
Invented scalar experts and router scores. Real experts are learned neural networks operating on vectors. This toy omits shared experts, residual additions, and load balancing; its renormalization is a teaching choice. More active experts does not automatically mean a better result.
Same operation, another internal update
Start at h = 0. Each step uses h ← h + a(4 − h). The number a stands in for a learned parameter; these are chosen scalar rules, not Transformer weights.
1distinct parameter sets
4applications of a rule
0tokens appended
We stop before any output head or token selection. Repeating a rule spends computation; convergence and better answers depend on the rule and training. Ordinary depth can also improve a representation.