Audio tokenization and codecs
Compress a waveform into audio tokens with an encoder, vector quantizer, residual codebooks, and decoder.
Language models predict discrete tokens—separate choices from a fixed vocabulary. A waveform is continuous: its values change smoothly through time. An audio codec bridges the two forms:
waveform → encoder → quantizer → token IDs → decoder → rebuilt waveform
This gives speech a vocabulary that a language model can predict.
Encode, choose, and rebuild
Section titled “Encode, choose, and rebuild”The encoder compresses the waveform into a shorter sequence of embeddings, or learned number-lists describing short sound moments.
The quantizer replaces each embedding with an allowed choice. The decoder uses those choices to rebuild the waveform.
The middle step is vector quantization (VQ). A codebook is a numbered collection of stored vectors. One stored vector is a codeword. VQ compares an encoder vector with the codewords, chooses the nearest one, and outputs its ID as a discrete audio token.
In the slide example, each codeword has 256 numbers and the codebook contains 1,024 choices. One codebook cannot capture all variation in speech: the result can sound distorted. Making one enormous codebook would slow the nearest-codeword search and leave many choices rarely used.
Correct what remains
Section titled “Correct what remains”Residual vector quantization (RVQ) uses several codebooks. The residual is the information the current codeword did not capture.
- Codebook 1 chooses a coarse sound vector.
- Subtract that choice from the encoder vector.
- Codebook 2 represents part of the remaining error.
- Later codebooks continue adding finer detail.
This is hierarchical refinement: broad structure first, detail afterward. The decoder adds the chosen codewords and turns the result back into audio. The first codebook is therefore the most important.
The EnCodec example uses 8 codebooks with 1,024 codewords each. Since 1,024 choices need 10 binary digits, or bits, each frame uses 8 × 10 = 80 bits of token IDs.
At a 24 kHz sampling rate, EnCodec produces 75 frames per second and 8 tokens per frame:
75 × 8 = 600 audio tokens per second
That is many more tokens than text—about 600 audio tokens while an average speaker says roughly 2.5 words—but still far fewer steps than predicting 24,000 waveform samples.
Train the codec
Section titled “Train the codec”A loss is a score describing what training should improve. The slides use three:
- Reconstruction loss compares rebuilt audio with the original.
- VQ loss keeps encoder vectors close to their selected codewords.
- Adversarial loss uses a learned judge to encourage realistic-sounding audio.
Key takeaways
Section titled “Key takeaways”- An audio codec converts continuous sound to discrete tokens and back.
- VQ chooses the nearest codeword from a codebook.
- RVQ uses several codebooks to add progressively finer detail.
- EnCodec’s example produces 600 audio tokens per second.