Fit, choose, then evaluate
- 1. Training
- 2. Validation
- 3. Final test
How can a gradient step fit the line?
Start with prediction a + bx at a = b = 0. Each step uses the training MSE gradient and learning rate 0.1. This demonstrates fitting; the candidate comparison uses completed fits.
Validation: choose among the fitted candidates
The new rows are (0, 1) and (3, 5).
Final test: evaluate the frozen line
The line remains 1 + x. Using this result to change the model would make these rows part of selection.
Compare with the population that generated these examples
We constructed the data using X uniform on 3 and Y = 1 + X + ε, with independent ε equally likely to be −1, 0, or 1. The fitting procedure did not receive this rule.
The line happens to equal the population mean in this example. Its expected squared error is (1 + 0 + 1)/3 = 2/3. The measured scores differ because each uses a few particular observations.