Fit the weights by least squares

Return to all three records and both features:

$$A=\begin{bmatrix}1&0\\2&1\\3&1\end{bmatrix}, \qquad y=\begin{bmatrix}2\\5\\8\end{bmatrix}.$$

No weights fit these scores exactly. Instead, choose weights that minimize the sum of squared residuals:

$$\widehat w=\operatorname*{argmin}_{w\in\mathbb R^2}\lVert y-Aw\rVert^2.$$

Here $y-Aw$ is the vector of observed minus predicted scores. Squaring its length adds the three squared errors. Argmin asks for the weights achieving the smallest value; it does not ask for the value of that error. The hat marks fitted weights.

From perpendicular error to equations

Every possible prediction lies in the column space of $A$. Least squares chooses the point in that space closest to $y$. The residual must therefore be perpendicular to each column, not merely to one of them:

$$A^\mathsf T(y-A\widehat w)=0.$$

Each entry of this product is a column dotted with the residual. Rearranging gives the normal equations:

$$A^\mathsf TA\widehat w=A^\mathsf Ty.$$

“Normal” here means perpendicular. Calculate the products:

$$\begin{bmatrix}14&5\\5&2\end{bmatrix} \begin{bmatrix}\widehat w_1\\\widehat w_2\end{bmatrix} =\begin{bmatrix}36\\13\end{bmatrix}.$$

For example, the top-left entry is $1^2+2^2+3^2=14$, and the top-right is $1(0)+2(1)+3(1)=5$. On the right, $1(2)+2(5)+3(8)=36$.

Multiply the first equation by 2 and the second by 5. Subtracting gives $3\widehat w_1=7$. Substitute back to obtain

$$\widehat w=\begin{bmatrix}7/3\\2/3\end{bmatrix}.$$

Check the fit

The predictions and residuals are

$$A\widehat w=\begin{bmatrix}7/3\\16/3\\23/3\end{bmatrix}, \qquad y-A\widehat w=\begin{bmatrix}-1/3\\-1/3\\1/3\end{bmatrix}.$$

The residual dotted with the hours column is $-1/3-2/3+3/3=0$. With the sets column it is $-1/3+1/3=0$. The squared errors sum to $1/3$. Dividing by three records gives training mean squared error $1/9$; this positive constant factor does not change the fitted weights.

The earlier weights $[2,1]^\mathsf T$ fit the first two records exactly but missed the third by 1, giving squared error 1. Least squares accepts smaller errors on all three records to reduce their total. It does not insist on preserving any individual exact fit.

Our independent columns make these weights unique. Dependent columns can give several weight vectors with the same closest prediction. Least squares itself does not require normally distributed noise; additional assumptions enter when interpreting uncertainty or making claims about the data-generating process.

This model has no intercept. Adding a column of ones would add one, changing the allowed predictions. A small training error also says nothing by itself about new records: that requires evaluation on held-out data.

Try it

What does the fitted model predict for a new input $x=[2,2]^\mathsf T$?

Work through the answer

$x^\mathsf T\widehat w=2(7/3)+2(2/3)=6$. This is a prediction from the fitted rule, not a known outcome or a causal estimate.

For more examples, see Georgia Tech’s least-squares chapter.