To learn meaning, hide a few words and guess them back — using the whole sentence, both directions.
original — what really was there
corrupted input — ~15% of tokens get a fate
the model's job — guess the original
[MASK]
10% → random word
10% → kept as-is
Only the ~15% chosen tokens count toward the loss. The random-swap and keep-as-is tricks stop the model from getting lazy — it can’t assume an untouched word is safe, so it has to understand every position.
Each "Mask again" re-rolls the dice — same sentence, new corruption.