Module 4 · Formula sheet

Back to the Module 4 guide.

Binary categories

Switch a group adjustment on or off. A dummy variable $D$ is 0 for the reference group and 1 for the other group:

$$\widehat Y=\widehat\beta_0+\widehat\beta_1D.$$

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the reference prediction; $\widehat\beta_1$ is the other group’s adjustment. Hats mark estimates fitted from data.

Substitute first:

$$\begin{aligned} D=0:&\quad\widehat Y=\widehat\beta_0,\\ D=1:&\quad\widehat Y=\widehat\beta_0+\widehat\beta_1. \end{aligned}$$

Invented example: $\widehat Y=40+15D$. Predictions are $40+15(0)=40$ and $40+15(1)=55$; the gap is 15.

With other predictors included, compare groups at the same input values. This constant gap assumes no interaction involving the group.

Multiple categories

One baseline; $K-1$ switches. $K$ is the number of categories. For nonreference category $j$, set $D_j=1$ when the observation belongs to it and 0 otherwise:

$$\widehat Y=\widehat\beta_0+ \sum_{j=1}^{K-1}\widehat\beta_jD_j.$$

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the fitted baseline; $\widehat\beta_j$ is category $j$‘s fitted adjustment. All switches are 0 for the reference.

$$\begin{aligned} \text{Reference prediction}&=\widehat\beta_0,\\ \text{Category }j\text{ prediction}&=\widehat\beta_0+\widehat\beta_j. \end{aligned}$$

For two nonreference categories $a,b$, the prediction difference is $\widehat\beta_a-\widehat\beta_b$.

Invented example: baseline 20, adjustments 5 and $-3$. Predictions: $20$, $20+5=25$, $20-3=17$. The last two differ by $5-(-3)=8$.

All $K$ switches would sum to the intercept column of ones, making the weights inseparable. Changing the reference changes coefficient meanings, but preserves predictions.

Quantitative interactions

Product = a new feature

For numerical inputs $X_1,X_2$, calculate $X_1X_2$ and give that column its own coefficient:

$$\begin{aligned} \widehat Y&=\widehat\beta_0+ \widehat\beta_1X_1+\widehat\beta_2X_2\\ &\quad +\widehat\beta_3X_1X_2. \end{aligned}$$

$\widehat Y$ is the prediction. The fitted coefficients are intercept $\widehat\beta_0$, main-input multipliers $\widehat\beta_1,\widehat\beta_2$, and product multiplier $\widehat\beta_3$.

Collect every term multiplying the input you change

Hold the other input fixed. One extra input unit changes the prediction by its slope:

$$\begin{aligned} \text{Slope for }X_1&=\widehat\beta_1+\widehat\beta_3X_2,\\ \text{Slope for }X_2&=\widehat\beta_2+\widehat\beta_3X_1. \end{aligned}$$

$\widehat\beta_1$ is the $X_1$ slope at $X_2=0$; $\widehat\beta_2$ is the $X_2$ slope at $X_1=0$. The interaction makes those slopes change.

Slide coefficients, invented input: TV coefficient 0.0191, product coefficient 0.0011, assumed radio input 20. TV’s slope is $0.0191+0.0011(20)=0.0411$.

Hierarchy principle: keep $X_1$ and $X_2$ when including $X_1X_2$, even if their individual tests are not significant.

Coefficient / standard error / t

Readout	Meaning
Coefficient	The fitted multiplier.
Standard error (SE)	Estimated sample-to-sample variation in that multiplier.
t-statistic	Gap from the tested value, measured in standard errors.

To test the true interaction coefficient $\beta_3=0$:

$$t=\frac{\widehat\beta_3-0} {\operatorname{SE}(\widehat\beta_3)}.$$

Under the module’s approximate two-sided rule, reject zero when $|t|>2$; otherwise fail to reject, including at $|t|=2$.

The interaction slide reports coefficient 0.0011 and $t=20.73$. Since $|20.73|>2$, reject zero under the model’s assumptions. Its displayed SE of 0.000 is rounded, not exactly zero; printed values need not reproduce the reported t. Statistical evidence alone proves neither causation nor better test predictions.

Exact t-tests

For an exact test under independent mean-zero normal errors of constant variance and linearly independent design columns, compare with a t-distribution having $n-p-1$ degrees of freedom. Here $n$ counts observations and $p$ counts input columns excluding the intercept; this interaction model has $p=3$.

Category interactions

The switch can add an intercept and a slope. Let $X$ be a numerical input, and $D=0$ for the reference group or $1$ for the other group:

$$\begin{aligned} \widehat Y&=\widehat\beta_0+ \widehat\beta_1X+\widehat\beta_2D\\ &\quad +\widehat\beta_3XD. \end{aligned}$$

$\widehat Y$ is the prediction. $\widehat\beta_0,\widehat\beta_1$ are the fitted reference intercept and slope. $\widehat\beta_2$ adds to the other group’s intercept; $\widehat\beta_3$ adds to its slope.

Substitute $D=0$ and $D=1$ before reading the lines:

$$\begin{aligned} D=0:\quad\widehat Y &=\widehat\beta_0+\widehat\beta_1X,\\ D=1:\quad\widehat Y &=(\widehat\beta_0+\widehat\beta_2)\\ &\quad +(\widehat\beta_1+\widehat\beta_3)X. \end{aligned}$$

At the same $X$, other-group prediction minus reference prediction is

$$\text{Group gap}=\widehat\beta_2+\widehat\beta_3X.$$

The lines are parallel when $\widehat\beta_3=0$.

Invented example: $\widehat Y=20+2X+10D+3XD$. The lines are $20+2X$ and $30+5X$. At $X=4$, predictions are 28 and 50; the gap rule gives $10+3(4)=22$.

Polynomial features

Calculate powers first; fit their weights second. For numerical input $X$, use columns $X,X^2,\ldots,X^k$:

$$\widehat Y=\widehat\beta_0+ \sum_{r=1}^{k}\widehat\beta_rX^r.$$

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the fitted intercept; $\widehat\beta_r$ multiplies power $r$. Degree $k$ is the largest power. The coefficients still multiply known columns and add: this is linear regression.

Invented example: for $\widehat Y=1+3X+2X^2$ at $X=2$, the row $[1,X,X^2]$ becomes $[1,2,4]$. Prediction: $1+3(2)+2(4)=15$.

Count the columns actually present. $p$ counts input columns excluding the intercept. A full cubic uses $[1,X,X^2,X^3]$: $p=3$. The specified model $\widehat Y=\alpha X^2+\gamma$ uses $[1,X^2]$: $p=1$ and two fitted coefficients, $\alpha$ and $\gamma$.

Training and test RSS

Circle TRAIN or TEST first. Training chooses the weights; test data checks those weights without refitting.

RSS is the total squared prediction error:

$$\mathrm{RSS}=\sum_{i=1}^{n}(y_i-\widehat y_i)^2.$$

$n$ counts observations, $y_i$ is actual output $i$, and $\widehat y_i$ is its prediction. Subtract, square, then add. Compare models on the same observations.

The cubic can copy the line

For ordinary least-squares fits with intercepts on the same training data, zeroing the cubic’s squared and cubed coefficients makes it a line. Therefore

$$\mathrm{RSS}_\text{cubic, train} \leq\mathrm{RSS}_\text{linear, train}.$$

This holds whatever generated the data; equality is possible.

True relationship	Test comparison
Linear	Expect the line to do better; individual datasets can differ.
Nonlinear, amount unknown	Not enough information to choose.

Training improvement gives no test guarantee: extra terms may learn real curvature or training noise.

Transformed least squares

Rename the column, reuse the line formula

To fit $\widehat y_i=\alpha x_i^2+\gamma$, calculate $z_i=x_i^2$. Here $x_i,y_i$ are supplied input and output for observation $i$; $\widehat y_i$ is its prediction; $\alpha$ is the weight and $\gamma$ the intercept to fit.

The model becomes $\widehat y_i=\alpha z_i+\gamma$. For $n$ observations, calculate means:

$$\bar z=\frac1n\sum_{i=1}^{n}z_i, \qquad \bar y=\frac1n\sum_{i=1}^{n}y_i.$$

Then calculate how the columns move together, $S_{zy}$, and the input spread, $S_{zz}$:

$$S_{zy}=\sum_{i=1}^{n}(z_i-\bar z)(y_i-\bar y),$$ $$S_{zz}=\sum_{i=1}^{n}(z_i-\bar z)^2.$$

For $S_{zz}>0$, the least-squares coefficients are

$$\boxed{\alpha=\frac{S_{zy}}{S_{zz}}}, \qquad \boxed{\gamma=\bar y-\alpha\bar z}.$$

If $S_{zz}=0$, every new input is identical and the slope cannot be uniquely identified.

Q6 shortcut: two distinct inputs

The supplied pairs $(0,0),(1,1/2),(-1,3/2)$ become $(0,0),(1,1/2),(1,3/2)$ in $(z,y)$ coordinates.

At $z=0$, the output mean is 0, so $\gamma=0$. At $z=1$, the output mean is $(1/2+3/2)/2=1$, so $\alpha+\gamma=1$ and $\alpha=1$. The fitted line hits both group averages, minimizing squared error.

Q6 through the general formula

With $n=3$, the means are $\bar z=(0+1+1)/3=2/3$ and $\bar y=(0+1/2+3/2)/3=2/3$.

Subtracting $2/3$ gives centered inputs $-2/3,1/3,1/3$ and centered outputs $-2/3,-1/6,5/6$.

$$\begin{aligned} S_{zy}&=(-\tfrac23)(-\tfrac23)\\ &\quad +(\tfrac13)(-\tfrac16)\\ &\quad +(\tfrac13)(\tfrac56)\\ &=\tfrac49-\tfrac1{18}+\tfrac5{18}=\tfrac23. \end{aligned}$$ $$\begin{aligned} S_{zz}&=(-\tfrac23)^2+(\tfrac13)^2+(\tfrac13)^2\\ &=\tfrac49+\tfrac19+\tfrac19=\tfrac23. \end{aligned}$$

Thus $\alpha=(2/3)/(2/3)=1$ and $\gamma=2/3-1(2/3)=0$.

Check: $\widehat Y=X^2$ predicts $0,1,1$ at original inputs $0,1,-1$. Subtract those predictions from supplied outputs $0,1/2,3/2$:

$$\mathrm{RSS}=0^2+(-\tfrac12)^2+(\tfrac12)^2=\tfrac12.$$

The general formula works for any specified new column $z_i=g(x_i)$, where $g$ is the rule for calculating it.