principles.fyi · the brain · concept

sparse autoencoder (SAE)

A tool that spreads tangled signals into a few clear, named features.

activations -> wide list (mostly 0s) -> rebuild activations

Inside a transformer, one number often does many jobs at once, so meanings get squished together. A sparse autoencoder is a small helper network that copies those numbers into a much wider list of slots, with a rule that almost all slots must stay quiet. Forced to use only a few slots at a time, it learns to give each one a single tidy meaning, like "talks about dogs" or "is in French." That lets us peek inside the model and read what it is actually thinking about.