Module 4 · Formula sheet

Back to the Module 4 guide.

Binary categories

Switch a group adjustment on or off. A dummy variable $D$ is 0 for the reference group and 1 for the other group:

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the reference prediction; $\widehat\beta_1$ is the other group’s adjustment. Hats mark estimates fitted from data.

Substitute first:

Invented example: $\widehat Y=40+15D$. Predictions are $40+15(0)=40$ and $40+15(1)=55$; the gap is 15.

With other predictors included, compare groups at the same input values. This constant gap assumes no interaction involving the group.

Multiple categories

One baseline; $K-1$ switches. $K$ is the number of categories. For nonreference category $j$, set $D_j=1$ when the observation belongs to it and 0 otherwise:

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the fitted baseline; $\widehat\beta_j$ is category $j$‘s fitted adjustment. All switches are 0 for the reference.

For two nonreference categories $a,b$, the prediction difference is $\widehat\beta_a-\widehat\beta_b$.

Invented example: baseline 20, adjustments 5 and $-3$. Predictions: $20$, $20+5=25$, $20-3=17$. The last two differ by $5-(-3)=8$.

All $K$ switches would sum to the intercept column of ones, making the weights inseparable. Changing the reference changes coefficient meanings, but preserves predictions.

Quantitative interactions

Product = a new feature

For numerical inputs $X_1,X_2$, calculate $X_1X_2$ and give that column its own coefficient:

$\widehat Y$ is the prediction. The fitted coefficients are intercept $\widehat\beta_0$, main-input multipliers $\widehat\beta_1,\widehat\beta_2$, and product multiplier $\widehat\beta_3$.

Collect every term multiplying the input you change

Hold the other input fixed. One extra input unit changes the prediction by its slope:

$\widehat\beta_1$ is the $X_1$ slope at $X_2=0$; $\widehat\beta_2$ is the $X_2$ slope at $X_1=0$. The interaction makes those slopes change.

Slide coefficients, invented input: TV coefficient 0.0191, product coefficient 0.0011, assumed radio input 20. TV’s slope is $0.0191+0.0011(20)=0.0411$.

Hierarchy principle: keep $X_1$ and $X_2$ when including $X_1X_2$, even if their individual tests are not significant.

Coefficient / standard error / t

To test the true interaction coefficient $\beta_3=0$:

Under the module’s approximate two-sided rule, reject zero when $|t|>2$; otherwise fail to reject, including at $|t|=2$.

The interaction slide reports coefficient 0.0011 and $t=20.73$. Since $|20.73|>2$, reject zero under the model’s assumptions. Its displayed SE of 0.000 is rounded, not exactly zero; printed values need not reproduce the reported t. Statistical evidence alone proves neither causation nor better test predictions.

Category interactions

The switch can add an intercept and a slope. Let $X$ be a numerical input, and $D=0$ for the reference group or $1$ for the other group:

$\widehat Y$ is the prediction. $\widehat\beta_0,\widehat\beta_1$ are the fitted reference intercept and slope. $\widehat\beta_2$ adds to the other group’s intercept; $\widehat\beta_3$ adds to its slope.

Substitute $D=0$ and $D=1$ before reading the lines:

At the same $X$, other-group prediction minus reference prediction is

The lines are parallel when $\widehat\beta_3=0$.

Invented example: $\widehat Y=20+2X+10D+3XD$. The lines are $20+2X$ and $30+5X$. At $X=4$, predictions are 28 and 50; the gap rule gives $10+3(4)=22$.

Polynomial features

Calculate powers first; fit their weights second. For numerical input $X$, use columns $X,X^2,\ldots,X^k$:

$\widehat Y$ is the prediction; $\widehat\beta_0$ is the fitted intercept; $\widehat\beta_r$ multiplies power $r$. Degree $k$ is the largest power. The coefficients still multiply known columns and add: this is linear regression.

Invented example: for $\widehat Y=1+3X+2X^2$ at $X=2$, the row $[1,X,X^2]$ becomes $[1,2,4]$. Prediction: $1+3(2)+2(4)=15$.

Count the columns actually present. $p$ counts input columns excluding the intercept. A full cubic uses $[1,X,X^2,X^3]$: $p=3$. The specified model $\widehat Y=\alpha X^2+\gamma$ uses $[1,X^2]$: $p=1$ and two fitted coefficients, $\alpha$ and $\gamma$.

Training and test RSS

Circle TRAIN or TEST first. Training chooses the weights; test data checks those weights without refitting.

RSS is the total squared prediction error:

$n$ counts observations, $y_i$ is actual output $i$, and $\widehat y_i$ is its prediction. Subtract, square, then add. Compare models on the same observations.

The cubic can copy the line

For ordinary least-squares fits with intercepts on the same training data, zeroing the cubic’s squared and cubed coefficients makes it a line. Therefore

This holds whatever generated the data; equality is possible.

Training improvement gives no test guarantee: extra terms may learn real curvature or training noise.

Transformed least squares

Rename the column, reuse the line formula

To fit $\widehat y_i=\alpha x_i^2+\gamma$, calculate $z_i=x_i^2$. Here $x_i,y_i$ are supplied input and output for observation $i$; $\widehat y_i$ is its prediction; $\alpha$ is the weight and $\gamma$ the intercept to fit.

The model becomes $\widehat y_i=\alpha z_i+\gamma$. For $n$ observations, calculate means:

Then calculate how the columns move together, $S_{zy}$, and the input spread, $S_{zz}$:

For $S_{zz}>0$, the least-squares coefficients are

If $S_{zz}=0$, every new input is identical and the slope cannot be uniquely identified.

Q6 shortcut: two distinct inputs

The supplied pairs $(0,0),(1,1/2),(-1,3/2)$ become $(0,0),(1,1/2),(1,3/2)$ in $(z,y)$ coordinates.

At $z=0$, the output mean is 0, so $\gamma=0$. At $z=1$, the output mean is $(1/2+3/2)/2=1$, so $\alpha+\gamma=1$ and $\alpha=1$. The fitted line hits both group averages, minimizing squared error.

Check: $\widehat Y=X^2$ predicts $0,1,1$ at original inputs $0,1,-1$. Subtract those predictions from supplied outputs $0,1/2,3/2$:

The general formula works for any specified new column $z_i=g(x_i)$, where $g$ is the rule for calculating it.