A fixed compute budget is split between model size and training data. Sliding the split makes one grow as the other shrinks. A loss curve below shows a valley: loss is worst when the budget is spent almost entirely on a big model with little data, or on a tiny model with lots of data, and best at the balanced split in the middle, which glows green.

Spend the compute budget wisely

Fixed compute. Bigger brain ↔ more books. There’s a sweet spot.

Balanced: roughly 20 tokens per parameter — the lowest loss for this budget.