Follow the state through 40 layers
Choose a component or layer. Layer numbers here start at 1. Within each stack, read left to right, then down; the original paper draws the flow upward.
01 · Causal encoder
Prepare representations of the prefix
02 · Decoder
Read prepared context and refine the state
Each tile = attention + MoE. All CSA2 tiles also contain local sliding-window attention. Colors identify modes, not shared layer weights.
How do different inputs become numbers the model can use?
Text and vision embeddings
Text tokens start with an embedding lookup. Images pass through a vision encoder and a projector into the language model’s representation space. Both enter the causal encoder as vectors. A vector is a list of numbers; it is not a written description of the image.
Token IDs or pixels → numerical representations
Technical report · §2.1 · Overview