Linear algebra reference

Objects and shapes

Use $n$ for records and $d$ for features. Our example has $n=3$ and $d=2$.

Expression	Meaning	Shape or type
$x_i$	Feature measurements for record $i$, as a column	$d\times1$
$x_{ij}$	Feature $j$ of record $i$	Scalar
$A$	Inputs, with record $i$ stored in row $i$	$n\times d$
$w$	One weight per feature	$d\times1$
$y$	Observed outputs	$n\times1$
$Aw$	One prediction per record	$n\times1$
$A^\mathsf T$	Rows and columns exchanged	$d\times n$
$y-Aw$	Residuals: observed minus predicted	$n\times1$

These shapes use the convention that feature vectors are columns. A dataset row is therefore $x_i^\mathsf T$.

Operations

$$u\cdot v=u^\mathsf Tv=\sum_{j=1}^d u_jv_j, \qquad \lVert v\rVert^2=v\cdot v.$$

The dot product returns one number. The squared Euclidean norm adds squared entries. Both depend on the chosen coordinate scales.

$$(m\times d)(d\times k)\longrightarrow(m\times k), \qquad (AB)_{ij}=\sum_{r=1}^{d} A_{ir}B_{rj}.$$

Check the inner dimensions, then take each left-hand row dotted with each right-hand column. Matrix multiplication is not entry-by-entry multiplication, and changing order can change the answer or make a product undefined.

Reachable outputs

If $a_1,\ldots,a_d$ are the columns of $A$, then

$$Aw=w_1a_1+\cdots+w_da_d.$$

The column space contains all these linear combinations. $Aw=y$ has an exact solution precisely when $y$ belongs to it. Independent columns give unique coefficients for every reachable output. Rank counts independent column directions.

Closest outputs under squared error

For a nonzero vector $a$, the projection onto its span is

$$p=a\frac{a^\mathsf Ty}{a^\mathsf Ta}, \qquad a^\mathsf T(y-p)=0.$$

For several columns, least-squares weights satisfy

$$\widehat w\in\operatorname*{argmin}_w\lVert y-Aw\rVert^2, \qquad A^\mathsf TA\widehat w=A^\mathsf Ty.$$

The fitted prediction is unique; the weights are unique if the columns are independent. Writing $(A^\mathsf TA)^{-1}A^\mathsf Ty$ requires that independence. Solving the normal equations does not require explicitly forming an inverse.

For our example:

$$\widehat w=[7/3,2/3]^\mathsf T, \quad \widehat y=[7/3,16/3,23/3]^\mathsf T, \quad \mathrm{SSE}=1/3, \quad \mathrm{MSE}=1/9.$$

An intercept requires a constant input column. Minimizing training error is a fitting calculation; assessing new-data performance is a separate step.