CTC: alignment without timestamps
Use CTC blanks and collapsing rules to learn the word “dinner” without labeling when each letter was spoken.
Suppose a recording becomes 300 sound frames, but its transcript is only “dinner”—six characters. Training gives the correct word, but not the exact frame where each character occurs. That missing map is the alignment problem.
Connectionist temporal classification (CTC) trains directly from audio and text without a hand-labeled frame-by-frame alignment.
Why simple repetition fails
Section titled “Why simple repetition fails”At every frame, the model predicts a distribution—a set of probability scores—over the alphabet. Speech lasts across many frames, so it may predict a letter repeatedly:
d i i n n n e r r r
Merging neighboring duplicates would produce “diner,” not “dinner.” The two written n characters need a separator.
Add the blank symbol
Section titled “Add the blank symbol”CTC adds the blank symbol, written here as ∅. Blank means “write nothing at this frame.” It can also sit between repeated letters.
The CTC collapsing function applies two rules in this order:
- Collapse neighboring duplicate symbols.
- Remove every blank.
For example:
d i ∅ n n ∅ n e r ∅
First merge the adjacent n n; then remove blanks. The result is “dinner.” Without the blank between the two n runs, they would merge into one letter.
The full frame-by-frame symbol list is one alignment, or proposed timing. Many different alignments can collapse to the same word. This is a many-to-one mapping: many paths, one transcript.
Train over every valid alignment
Section titled “Train over every valid alignment”At each frame, CTC predicts a probability for every letter plus blank. The probability of one alignment is the product of its frame probabilities. The probability of “dinner” is the sum of the probabilities of every alignment that collapses to “dinner.”
CTC loss = −log(probability of the correct transcript)
The negative logarithm is only a converter: high probability for the correct transcript becomes a small loss, while low probability becomes a large loss.
There are exponentially many possible alignments, so listing them would be too slow. The forward–backward algorithm uses dynamic programming—saving and reusing partial results—to add them efficiently. With T frames and |Y| transcript symbols, the slide gives O(T × |Y|) time: the work grows roughly with frames multiplied by transcript length.
Read the model’s output
Section titled “Read the model’s output”Greedy decoding chooses the highest-scoring symbol at each frame and then collapses the path. Beam search keeps several possible outputs before choosing one.
CTC can also be trained beside an encoder–decoder attention loss. The two training scores give the encoder two useful learning signals. For live transcription, RNN-Transducer (RNN-T) combines the current audio state with earlier output tokens and is designed for streaming ASR.
Key takeaways
Section titled “Key takeaways”- CTC learns without explicit frame-level alignments.
- Blank separates repeated characters.
- Many alignments can collapse to the same transcript.
- Forward–backward sums those alignments efficiently.