HuBERT groups sound frames, masks some frames, and predicts their cluster labels before fine-tuning for speech recognition.
HuBERT learns before transcripts
Make numbered sound groups, hide some frames, and predict their group labels from nearby audio. Then fine-tune the model for ASR.
pretraining uses unlabeled audio—recordings without transcripts
1 · make cluster labels from the audio
target row · each number names one sound cluster
2 · hide three frames · [MASK] means hidden
CNN encoder + transformerThe CNN makes frame features. The transformer uses sound on both sides of the mask.
left sound → mask ← right sound
Self-supervised learning learns from unlabeled audio. k-means groups similar feature lists and gives each group a discrete label. A mask hides selected frames. HuBERT learns by predicting the hidden cluster labels, then uses transcripts only during ASR fine-tuning.