Module 3 formulas

Back to Module 3.

Matrix products

Inner dimensions match; outer dimensions survive.

For real matrices $A$ of shape $m\times n$ and $B$ of shape $n\times p$:

$$(m\times n)(n\times p)\longrightarrow(m\times p),$$ $$(AB)_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}.$$

• $i$: output row; $j$: output column.
• $k$: index along the matched row and column.
• $A_{ik},B_{kj}$: individual entries.
• $\sum$: add the products as $k$ runs from 1 to $n$.

One entry = row times column. Row $[1,2]$ and column $[2,1]^\mathsf T$ give $1(2)+2(1)=4$.

Powers and identity

$A^2=AA$; $A^3=(AA)A$. The identity $I$ has diagonal entries 1 and other entries 0: $AI=IA=A$ when sizes match.

Algebra rules

• Generally $AB\ne BA$.
• $(AB)C=A(BC)$.
• $A(B+C)=AB+AC$.

The transpose $A^\mathsf T$ swaps rows and columns:

$$(A^\mathsf T)_{ij}=A_{ji}, \qquad (A^\mathsf T)^\mathsf T=A.$$ $$(A+B)^\mathsf T=A^\mathsf T+B^\mathsf T,$$ $$(AB)^\mathsf T=B^\mathsf TA^\mathsf T.$$

Transposing a product reverses its order. All operations require compatible dimensions.

Vector products

Inner = one number; outer = a grid. Let $u,v$ be column vectors with $n$ real entries.

$$u^\mathsf Tv=\sum_{k=1}^n u_kv_k=v^\mathsf Tu,$$ $$(uv^\mathsf T)_{ij}=u_iv_j.$$

$u_k,v_k$ are vector entries. The outer product is an $n\times n$ matrix; $i,j$ select its row and column.

Matrix times vector: for $A$ with $m$ rows and $n$ columns,

$$(Av)_i=\sum_{k=1}^n A_{ik}v_k.$$

The result has $m$ entries. Each is one row of $A$ dotted with $v$.

Squared length = sum of squared entries:

$$\lVert v\rVert^2=v^\mathsf Tv=\sum_{k=1}^n v_k^2.$$

Invented example: $u=[2,1]^\mathsf T$, $v=[3,4]^\mathsf T$ give inner product $2(3)+1(4)=10$ and outer product $\begin{bmatrix}6&8\\3&4\end{bmatrix}$.

Eigenvectors

Multiply, then check for one common scaling factor.

$$Mv=\lambda v,\qquad v\ne0.$$

• $M$: square $n\times n$ matrix.
• $v$: nonzero eigenvector.
• $\lambda$: eigenvalue, the multiplier.

Negative $\lambda$ reverses direction; zero collapses the vector to zero. For invented $M=\begin{bmatrix}3&0\\0&2\end{bmatrix}$ and $v=[1,0]^\mathsf T$, $Mv=[3,0]^\mathsf T=3v$.

If $M$ is real and symmetric, $M=M^\mathsf T$, its eigenvalues are real and one can choose $n$ mutually perpendicular eigenvectors. Perpendicular means $v_i^\mathsf Tv_j=0$ for different chosen vectors. General real matrices may have complex eigenvalues/eigenvectors.

Regression model

Prediction = baseline + input contributions.

$$Y=\beta_0+\sum_{j=1}^{p}\beta_jX_j+\varepsilon.$$

• $Y$: response; $X_j$: input $j$.
• $p$: number of predictors, excluding the intercept.
• $\beta_0$: intercept, the baseline when all inputs are zero.
• $\beta_j$: multiplier, or response change per unit of input $j$, holding other inputs fixed.
• $\varepsilon$: unexplained noise.

A hat means “estimated from the data.” For observation $i$ with recorded inputs $x_{ij}$:

$$\widehat y_i=\widehat\beta_0+\sum_{j=1}^{p}\widehat\beta_jx_{ij}.$$

Unknown noise is left out of the prediction. Invented rule $\widehat y=10+3x$ at $x=4$: contribution $3(4)=12$; add baseline $10+12=22$.

Probability story: draw inputs, then draw an output given those inputs.

$$f_{X,Y}(x,y)=f_{Y\mid X}(y\mid x)f_X(x).$$

$f_X$ is the input density, $f_{Y\mid X}$ the conditional output density, and $f_{X,Y}$ their joint density. $x,y$ are possible input and output values; the model plus noise distribution describes the conditional output.

Simple regression

Center → multiply → add → divide. Given $n$ observed pairs $(x_i,y_i)$, fit $\widehat y=\widehat\beta_0+\widehat\beta_1x$ with an intercept.

1. Means

$$\bar x=\frac1n\sum_{i=1}^n x_i, \qquad\bar y=\frac1n\sum_{i=1}^n y_i.$$

2. Centered totals

$$S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y),$$ $$S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2.$$

$S_{xy}$ measures joint movement; $S_{xx}$ measures input spread.

3. Slope and intercept

$$\widehat\beta_1=\frac{S_{xy}}{S_{xx}}, \qquad\widehat\beta_0=\bar y-\widehat\beta_1\bar x.$$

Require $S_{xx}>0$: identical inputs cannot identify a slope.

The intercept puts the line through the averages:

$$\begin{aligned} \widehat y(\bar x) &=(\bar y-\widehat\beta_1\bar x)+\widehat\beta_1\bar x\\ &=\bar y. \end{aligned}$$

Shared Q3: pairs $(-1,-2),(0,0),(1,5)$ give calculated means $0,1$, $S_{xx}=2$, and $S_{xy}=7$. Thus slope $=7/2=3.5$; intercept $=1-3.5(0)=1$.

Matrix least squares

One matrix fits every row at once.

• $M$: design matrix with $n$ rows and $p+1$ columns; first column all ones, then $p$ predictor columns.
• $y$: column of $n$ observed outputs.
• $\theta$: trial vector of $p+1$ coefficients.
• $\widehat\beta$: fitted coefficient vector, intercept first.

$$\widehat\beta=\operatorname*{argmin}_{\theta} \lVert y-M\theta\rVert^2.$$

Argmin selects the coefficients with smallest RSS. The squared length adds squared residual entries.

With linearly independent columns — no column an exact combination of others — use

$$\widehat\beta=(M^\mathsf TM)^{-1}M^\mathsf Ty,$$ $$\widehat y=M\widehat\beta.$$

$\mathsf T$ means transpose. An inverse undoes a matrix: $A^{-1}A=I$, with identity $I$. The displayed inverse requires independent design columns.

Fit metrics

Choose the right comparison. For $n$ observed outputs $y_i$, predictions $\widehat y_i$, and sample mean $\bar y=\frac1n\sum_i y_i$:

Residual = observed − predicted

$$e_i=y_i-\widehat y_i.$$

RSS = fitted model’s squared misses

$$\mathrm{RSS}=\sum_{i=1}^{n}(y_i-\widehat y_i)^2.$$

TSS = mean-only guess’s squared misses

$$\mathrm{TSS}=\sum_{i=1}^{n}(y_i-\bar y)^2.$$

R² = fraction of baseline error removed

$$R^2=1-\frac{\mathrm{RSS}}{\mathrm{TSS}} =\frac{\mathrm{TSS}-\mathrm{RSS}}{\mathrm{TSS}}.$$

Require TSS $>0$. For training-set least squares with an intercept, RSS $\le$ TSS and $0\le R^2\le1$.

RSE = estimated noise size in output units

$$\mathrm{RSE}=\sqrt{\frac{\mathrm{RSS}}{n-p-1}}.$$

$p$ counts predictor coefficients, excluding the intercept. Require $n>p+1$ and linearly independent design columns. RSS/TSS have squared output units; $R^2$ has no units.

Shared Q3: RSS $=1.5$, TSS $=26$, $n=3$, $p=1$. Then RSE $=\sqrt{1.5/(3-1-1)}=\sqrt{1.5}$ and $R^2=1-1.5/26\approx0.94$.

Coefficient uncertainty

RSE describes output misses; SE describes estimate spread. Smaller coefficient SE means a more precise estimate.

Assumptions: correct linear model $y=M\beta+\varepsilon$; zero-mean errors given inputs; independent errors with common variance $\sigma^2$; linearly independent design columns.

• $M$: design matrix, including an intercept.
• $y$: observed-output vector; $\varepsilon$: error vector.
• $\beta$: true coefficient vector; $\widehat\beta$: least-squares estimate.
• $\sigma$: noise standard deviation; if variance $\sigma^2$ is supplied, take its square root.

Unbiased = centered on the truth

$$\mathbb E[\widehat\beta\mid M]=\beta.$$

$\mathbb E$ averages estimates across repeated samples with design $M$ fixed. This does not make every estimate close.

Covariance = joint uncertainty

$$\operatorname{Cov}(\widehat\beta\mid M) =\sigma^2(M^\mathsf TM)^{-1}.$$

Its diagonal entries are coefficient variances. Their square roots are standard deviations (SDs).

One-predictor formulas

For $n$ inputs $x_i$, define $\bar x=\frac1n\sum_i x_i$ and $S_{xx}=\sum_i(x_i-\bar x)^2>0$.

$$\operatorname{SD}(\widehat\beta_1) =\frac{\sigma}{\sqrt{S_{xx}}},$$ $$\operatorname{SD}(\widehat\beta_0) =\sigma\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}.$$

$\beta_1$ is the slope; $\beta_0$ is the intercept.

Unknown noise? Replace $\sigma$ with RSE:

$$\operatorname{SE}(\widehat\beta_1) =\frac{\mathrm{RSE}}{\sqrt{S_{xx}}},$$ $$\operatorname{SE}(\widehat\beta_0) =\mathrm{RSE}\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}.$$

For one predictor, RSE $=\sqrt{\mathrm{RSS}/(n-2)}$, where RSS $=\sum_i(y_i-\widehat y_i)^2$ uses observed outputs $y_i$ and predictions $\widehat y_i$.

Calculation chain: RSS → RSE → coefficient SE. If the problem supplies a coefficient SE, use it directly.

Invented example: RSE $=1$ and $S_{xx}=10$ give slope SE $=1/\sqrt{10}\approx0.316$.

Intervals and tests

Interval = estimate ± margin. Test statistic = gap ÷ uncertainty.

• $\widehat\beta_j$: fitted coefficient $j$.
• $s_j=\operatorname{SE}(\widehat\beta_j)>0$: its standard error.
• $b$: proposed true coefficient value.

Quick approximately 95% interval

$$[\widehat\beta_j-2s_j,\ \widehat\beta_j+2s_j].$$

Calculate margin $2s_j$, then subtract/add it. Across repeated samples, about 95% of these intervals contain the fixed true coefficient $\beta_j$, under the normal approximation.

Two-sided test

$$H_0:\beta_j=b, \qquad H_a:\beta_j\ne b,$$ $$t=\frac{\widehat\beta_j-b}{s_j}.$$

$H_0$ is the null claim; $H_a$ is the alternative. $|t|$ is the gap measured in SEs.

• $|t|>2$: reject under the approximate rule.
• $|t|\le2$: fail to reject; the value remains consistent with this test.

Equivalently, reject values outside the interval. Endpoints count as inside for the practice convention. Failure to reject does not prove the null true.

Shared Q5(a): estimate $-3$, SE $1$, claim $b=1$ give $t=(-3-1)/1=-4$. The gap is 4 SEs: reject. For estimate 2 and SE $1/2$, the interval is $[1,3]$, so neither 1 nor 2 is rejected under this convention.

Zero-slope question: use $j=1$, $b=0$. Significance $\alpha\approx0.05$ describes the repeated-test rejection rate when the null is true, not the probability the null is true.

Approximation: 2 rounds the normal cutoff 1.96. Under a correct linear model with independent, zero-mean normal errors of constant variance, estimated noise, and independent design columns, exact 95% intervals replace 2 with $t_{0.975,n-p-1}$. This is the Student $t$ cutoff with $n-p-1$ degrees of freedom; $n$ counts observations and $p$ predictors excluding the intercept. The practice uses the approximate rule.