Where the error comes from

The previous decomposition scored one fixed predictor. Now ask what happens if you repeat the entire training process with different data.

The fit is random too

Draw a training dataset $D$ and fit a model $\hat f_D$. Draw a different dataset from the same population and the fitted model usually changes. At one fixed input $x_0$, those models give different predictions.

There are now two sources of randomness: which training dataset you got, and which future output you are asked to predict.

experiment	keep fixed	vary
possible outputs at $x_0$	the input	the fresh output $Y_0$
predictions at $x_0$	the input and training method	the training dataset $D$

The second experiment defines the average prediction $\mathbb E_D[\hat f_D(x_0)]$ and the prediction variance $\operatorname{Var}_D(\hat f_D(x_0))$. The subscript $D$ tells you what the averaging is over.

The displayed averages and variances are estimates from simulated datasets. A small collection of fits can have an average far from its theoretical expectation, especially near the edges.

Try it

Compare degrees 1, 3, and 20. Move x₀ toward an edge. Read the numbers before the explanation: do the predictions agree, and is their average close to the truth? Generate new datasets and see whether your conclusion survives.

The same algebra, twice

Let $\bar f(x_0)=\mathbb E_D[\hat f_D(x_0)]$. This is the population average prediction over hypothetical repeated training datasets—not the mean of the observed outputs.

At $x_0$, define bias as that average prediction minus the true conditional mean:

$$\operatorname{Bias}(x_0)=\bar f(x_0)-f(x_0).$$

For each fixed dataset, the earlier result gives expected error $(f(x_0)-\hat f_D(x_0))^2+\sigma^2$. Now average that expression over datasets. Apply the same variance identity to the random prediction $\hat f_D(x_0)$:

$$\begin{aligned} R(x_0)&=\mathbb E_{D,Y_0\mid x_0}[(Y_0-\hat f_D(x_0))^2]\\ &=\underbrace{(\bar f(x_0)-f(x_0))^2}_{\text{squared bias}}\\ &\quad+\underbrace{\operatorname{Var}_D(\hat f_D(x_0))}_{\text{prediction variance}}+\underbrace{\sigma^2}_{\text{output noise}}. \end{aligned}$$

We assume the new output is independent of the training data, conditional on the fixed input. Its zero-mean noise therefore contributes no cross term. If output variance depends on the input, replace $\sigma^2$ with $\sigma^2(x_0)$.

Bias asks whether the average prediction misses. Variance asks whether the fitted models disagree. Noise asks how much future outputs vary even at the same input.

Put numbers on the three parts

For a small toy experiment, suppose a training method produces each of the five listed predictions with equal probability. The true conditional mean is 4 and the output noise variance is 1.

predictions	mean	bias	bias²	prediction variance	expected error
7, 7, 8, 6, 7	7	3	9	0.4	10.4
4, 5, 4, 3, 4	4	0	0	0.4	1.4
1, 7, 4, 6, 2	4	0	0	5.2	6.2

For the last row, variance is $(9+9+0+4+4)/5=5.2$. The average prediction is correct, but an individual fit can still miss badly.

For a real simulation with only five sampled fits, these numbers would be estimates of the theoretical quantities. More simulation runs can improve those estimates; that is different from adding more training examples to each fit.

Expected error is different from one test-set score

$R(x_0)$ averages over training datasets and future outputs at a particular input. To obtain overall expected error, also average over inputs: $R=\mathbb E_X[R(X)]$.

A measured test MSE uses one fitted model and one finite test set. It is a noisy estimate of that fitted model’s performance. It is not exactly equal to the bias–variance decomposition across hypothetical datasets.

The tradeoff, and the five curves

In the classical example here, a rigid line cannot follow a curved relationship, while a high-degree polynomial can respond strongly to the noise in a small training set. This often produces decreasing bias, increasing prediction variance, and a U-shaped expected-error curve.

The decomposition is an identity under the stated conditions. Those particular curve shapes are a pattern, not a theorem about every learning method. Regularization, different model families, and sufficiently large models can produce different patterns. The advanced example called double descent is one reason to avoid treating the U as universal.

A schematic classical example, not measurements from the simulation above. Explain the mechanism behind each curve rather than memorizing its color or number.

Rigid or flexible?

These are starting hypotheses to check with validation data:

situation	a reasonable starting point	reasoning
many observations, few features	allow more flexibility	more data can stabilize a flexible fit
many features, few observations	constrain or regularize the fit	otherwise many very different models may fit the same data
strongly nonlinear relationship	allow a suitable nonlinear shape	an inappropriate straight line can retain approximation error
large output noise	use stronger smoothing or regularization	small sample fluctuations are less reliable

More data often reduces prediction variance. It does not make a straight-line model capable of representing every curved relationship. However, an estimator’s bias can also depend on sample size; “more data never changes bias” is too broad.

Practice and reference

Read an unfamiliar problem, review the cheat sheet, or try mixed practice.