Grade it on dots it hasn't seen

A model can fit the examples you gave it while making poor predictions on new examples. To discover that, separate the data used for fitting from the data used for evaluation.

Three jobs for data

dataset	its job	what you can choose with it
Training	fit parameters, such as slopes	parameter values
Validation	compare fitted candidates	degree, model family, regularization
Test	assess the final choice	nothing further in this experiment

For a fixed model $\hat f$ and $M$ held-out examples, the measured mean squared error is

$$\operatorname{MSE}_{\mathrm{held\ out}}=\frac{1}{M}\sum_{i=1}^{M}(y_i-\hat f(x_i))^2.$$

Read it as a loop: take example $i$, predict its output, subtract from the observed output, square, repeat, and divide the total by the number of examples. This is a calculation on a finite dataset.

The training formula is identical, but uses the examples that fitted the model. Whether a held-out set is called validation or test depends on how you use its result. If you use it to pick the degree, it is serving as validation data.

Why minimum training error cannot increase

A degree-4 polynomial can reproduce any degree-3 polynomial by setting its fourth-degree coefficient to zero. So the best degree-4 fit cannot have a higher training MSE than the best degree-3 fit.

The conditions matter: we are fitting the same data, minimizing the same unpenalized squared loss, over nested model families, and finding the minimum. An imperfect optimizer or a changed training objective need not produce this monotone sequence. “More parameters” on its own is not a proof.

Fresh examples reveal what the fit learned

Filled points fit each candidate; hollow points help choose its degree. The held-out points here are validation data. A separate final test set would assess the selected model.

Try it

Compare degrees 1, 3, and 20. Then generate new examples. Which degree would you choose from this validation set? Does that choice stay the same after resampling? The exact curve is noisy; a smooth U is not guaranteed on every sample.

A rigid model may miss a curved relationship: underfitting. A flexible model may track peculiarities of the training sample that do not repeat: overfitting. Compare errors against the noise level and reasonable alternatives; “both errors are high” alone is not a diagnosis, since the task itself may be noisy.

It is possible for measured training or validation MSE to be below $\sigma^2$. The noise floor applies to expected squared prediction error, not to every finite set of outcomes. Low training error alone does not prove overfitting; poor generalization is what matters.

Why keep a final test set?

If you try many models and pick the best validation score, you have partly adapted your choice to that set’s random quirks. Its winning score can therefore be optimistic. Evaluate the frozen choice on separate test data, and report that result without using it to retune.

For a small dataset, cross-validation reuses different training/validation splits to help select the model. Keep a final test set when feasible, or use nested cross-validation for assessment. The evaluation guide explains the distinction.

For these introductory examples, observations are independent draws from the same population. Time series, repeated measurements of the same person, or future deployment on a changed population require a split that respects that structure.

Next

Why can a method’s performance change across training samples? Separate bias, prediction variance, and output noise.