principles.fyi · the brain · concept

transfer learning

Learn general skills once on tons of text, then reuse them for many specific jobs.

pretrain on lots of text -> fine-tune on a little task data

Transfer learning is the two-stage recipe behind encoders: first pretrain on huge unlabeled text (via masked language modeling) to learn language in general, then transfer that knowledge to a specific task by fine-tuning on a small labeled dataset. It's the same pretrain-then-fine-tune split Book 01 used, and it's powerful because the expensive, knowledge-building step is done once and shared, so each new task only needs a little data and a small added head.