Midterm practice · Formula sheet

Back to the practice module

Use the question’s probability event, units, and noise assumptions before substituting numbers. In $N(\mu,\sigma^2)$, the second parameter is variance. A hat marks an estimate or prediction.

Bayes rule

Use for Problems 1, 3, 4. To minimize expected mistakes with equal error costs, choose the most probable class at each input.

Let $p_k(x)=P(Y=k\mid X=x)$ be the conditional probability of class $k$ at predictor value $x$. For multiple classes, $C_B(x)=\arg\max_k p_k(x)$: the label with largest probability. For two classes,

Either class is optimal at a tie; this module uses 1. If given $p_0(x)$ instead, predict 1 when $p_0(x)\le1/2$. Respect any stated domain. Check: $p_1=0.7$ gives prediction 1 and conditional error $1-0.7=0.3$.

With false-positive cost $c_{FP}>0$ and false-negative cost $c_{FN}>0$, assuming correct decisions cost zero, predict 1 when

Why: the expected cost of predicting 1 is $c_{FP}(1-p_1)$; predicting 0 costs $c_{FN}p_1$. Choose the smaller. Invented check: costs 1 and 4 give a threshold $1/(1+4)=0.2$.

Classification errors

Use for Problems 3, 4. Positive means actual or predicted class 1. $TP$ counts correct positives; $FN$ counts positives predicted 0; $FP$ counts negatives predicted 1; $TN$ counts correct negatives.

Rates require a nonzero actual-class denominator. Sheet check: $TP=4,FN=3,FP=2,TN=4$ gives $\mathrm{FNR}=3/7$ and $\mathrm{FPR}=2/6=1/3$.

For a sigmoid probability, $\sigma(z)=1/(1+e^{-z})$ is increasing and $\sigma(0)=1/2$. Thus the decision boundary is $z=0$. The favored class is the event inside the supplied probability. In Problem 3, $P(Y=0\mid x)=\sigma(2+x_1-x_2)$, so $z>0$ predicts 0, and the boundary is $x_2=x_1+2$.

Flipping all predictions swaps the two columns of the table; it does not change actual class totals.

Expectation and dependence

Use for Problem 4. If $Y=h(X)+\varepsilon$ with $\varepsilon$ independent of $X$, $E[\varepsilon]=\mu$, and $\operatorname{Var}(\varepsilon)=\sigma^2$, then

Here $h$ is the supplied deterministic rule, $f$ is the best squared-error prediction, and the last quantity is irreducible MSE. Sheet check: $h(x)=x+x^2$ and $N(1,2)$ noise give $f(x)=x+x^2+1$ and irreducible MSE 2. A marginal noise distribution alone does not imply independence.

For random variables $U,V$ with finite, nonzero variances, let $s_U,s_V$ be their standard deviations. Then

Independence implies zero covariance. Zero covariance does not imply independence. Counterexample: $X$ uniform on $\{-1,0,1\}$ and $Z=X^2$ are dependent, but $E[X]=E[XZ]=0$, so their covariance and correlation are zero. Correlation alone also does not establish causation.

Model flexibility

Use for Problems 4, 6. To measure new-data error, use predictions on observations that were not used for fitting. For $m$ evaluation pairs $(x_i,y_i)$ and fitted rule $\hat f$,

Select flexibility using validation MSE or cross-validation; reserve a final test set for evaluation after selection. Invented check: validation MSEs 4, 2, 3 for increasing flexibility favor the middle model.

At fixed input $x$, for a new independent response $Y$ and fitted model $\hat f_D$ trained on random dataset $D$,

Here $f(x)=E[Y\mid X=x]$, $\operatorname{Bias}(x)=E_D[\hat f_D(x)]-f(x)$, and $\sigma^2(x)=\operatorname{Var}(Y\mid X=x)$ is irreducible noise. The expectation averages over training datasets and fresh response noise. Overfitting by a very flexible model is associated with high variance, not high bias.

When an enlarged model contains the old model and both minimize ordinary training RSS on the same data,

It can match the old model by setting new coefficients to zero. Equality is possible. There is no corresponding general guarantee for test RSS or for an arbitrary subgroup’s RSS; Problem 6’s unchanged male fit follows from that model’s separate group parameters.

Coin flips

Use for Problem 5. Assume $n$ independent flips, each with heads probability $p$. With $h$ heads and $n-h$ tails,

The binomial coefficient $\binom nh=n!/[h!(n-h)!]$ counts choices of head positions. Factorial $n!=n(n-1)\cdots1$, with $0!=1$.

Sheet check: $P(H,H,T)=p^2(1-p)$; 20 heads in 30 flips has probability $30{,}045{,}015\,p^{20}(1-p)^{10}$. Use the counting factor only when different orders qualify.

ROC and AUC

Use for Problem 8. Let $Y$ be the true label and $\hat Y$ the prediction. An ROC curve plots

as a threshold or classifier parameter varies. AUC is the area under this curve in the unit square.

If an independent coin predicts 1 with probability $p$, both conditional probabilities are $p$:

Varying $p$ from 0 to 1 traces the diagonal, whose triangular area is $\mathrm{AUC}=1/2$. Check: $p=0.7$ gives $(0.7,0.7)$ and FNR $1-0.7=0.3$. A fixed $p$ is one point; the family gives the curve. With no label-related information, population AUC cannot systematically rise above or fall below $1/2$; sample estimates can fluctuate by chance.

Least squares · Fit an intercept and slope

Use this for a straight-line prediction $\hat y=\hat\beta_0+\hat\beta_1x$. The intercept $\hat\beta_0$ is the fitted response at $x=0$; the slope $\hat\beta_1$ is the change in the fitted response for a one-unit increase in $x$.

For $n$ supplied points $(x_i,y_i)$, calculate the means:

$\bar x=\frac1n\sum_{i=1}^n x_i,\qquad \bar y=\frac1n\sum_{i=1}^n y_i.$

Measure horizontal spread and joint variation:

$S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2$

$S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y).$

When $S_{xx}>0$, the coefficients minimizing squared residuals are:

$\hat\beta_1=\frac{S_{xy}}{S_{xx}}$

$\hat\beta_0=\bar y-\hat\beta_1\bar x.$

Use it. Problem 2 gives $S_{xy}=16$, $S_{xx}=10$, $\bar x=1$, and $\bar y=0$. Thus $\hat\beta_1=16/10=1.6$ and $\hat\beta_0=0-(1.6)(1)=-1.6$.

The fitted line estimates the mean response. If the model’s noise has known nonzero mean $m$, the fitted intercept estimates $\beta_0+m$; see shifted noise.

Fit quality · RSS, RSE, and R²

For observation $i$, the residual is $e_i=y_i-\hat y_i$, where $y_i$ is observed and $\hat y_i$ is fitted. With $n$ observations:

$\mathrm{RSS}=\sum_{i=1}^n e_i^2.$

RSS is total squared prediction error on the observations being evaluated. For a simple linear model with a fitted intercept and slope, the residual standard error is:

$\mathrm{RSE}=\sqrt{\frac{\mathrm{RSS}}{n-2}}.$

The denominator uses $n-2$ residual degrees of freedom because two coefficients were estimated. RSE has the same units as the response.

Let $\bar y$ be the observed response mean. The total sum of squares is the error from always predicting that mean:

$\mathrm{TSS}=\sum_{i=1}^n(y_i-\bar y)^2.$

For $\mathrm{TSS}>0$, the fraction of baseline squared error removed by the fit is:

$R^2=1-\frac{\mathrm{RSS}}{\mathrm{TSS}}.$

Use it. Problem 2 has $\mathrm{RSS}=0.4$, $n=5$, and $\mathrm{TSS}=26$. Thus $\mathrm{RSE}=\sqrt{0.4/3}\approx0.3651$ and $R^2=1-0.4/26\approx0.9846$.

Coefficient uncertainty · Known Gaussian noise variance

Assume $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, with fixed observed $x_i$ and independent errors $\varepsilon_i\sim N(0,\sigma^2)$. Here $\sigma>0$ is the supplied noise standard deviation, $n$ is the observation count, $\bar x$ is the predictor mean, and $S_{xx}=\sum_i(x_i-\bar x)^2>0$.

The standard errors describe how much the fitted coefficients vary over repeated samples:

$\mathrm{SE}(\hat\beta_1)=\frac{\sigma}{\sqrt{S_{xx}}}$

$\mathrm{SE}(\hat\beta_0)=\sigma\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}.$

For either coefficient $\beta_j$ ($j=0$ for intercept, $j=1$ for slope), a 95% confidence interval is:

$\hat\beta_j\pm1.96\,\mathrm{SE}(\hat\beta_j).$

The normal multiplier 1.96 is rounded. Over repeated samples, this interval-building procedure covers the true fixed coefficient about 95% of the time.

Use it. Problem 2 supplies variance $8s^2$, so $\sigma=\sqrt8\,s$. With $S_{xx}=10$, the slope standard error is $\sqrt{4/5}\,s$. Its estimate is 1.6, so its interval is $1.6\pm1.96\sqrt{4/5}\,s$.

If noise variance is instead unknown and estimated from residuals, an exact small-sample interval uses the appropriate $t$ multiplier and estimated standard errors. Problems 2 and 7 request intervals as functions of the supplied noise parameters, so use their given variances here.

Hypothesis tests · Compare a proposed coefficient with its interval

For a two-sided 5% test of $H_0:\beta_j=b$, $b$ is the proposed coefficient, $\hat\beta_j$ is its estimate, and $\mathrm{SE}(\hat\beta_j)$ is its standard error. Under independent Gaussian errors with known variance, use:

$z=\frac{\hat\beta_j-b}{\mathrm{SE}(\hat\beta_j)}.$

Reject when $|z|>1.96$. Equivalently, reject when the proposed value lies outside the 95% interval:

$|\hat\beta_j-b|>1.96\,\mathrm{SE}(\hat\beta_j).$

Fail to reject when the inequality is reversed or equal. This means there is insufficient evidence against the proposed value at this level; it does not prove the null hypothesis.

Use it. In Problem 7, the slope estimate is 2, the proposed slope is 0, and its standard error is $w/\sqrt{10}$. Reject when $2>1.96w/\sqrt{10}$, or $0<w<2\sqrt{10}/1.96\approx3.2268$.

Interactions · Substitute the group indicator

Let $x$ be a numeric predictor and $g\in\{0,1\}$ a group indicator. In the fitted model:

$\hat y=b_0+b_1x+b_2g+b_3xg,$

the $b_j$ are fitted coefficients. Substitute the group value to read its line:

$g=0:\quad \hat y_0=b_0+b_1x$

$g=1:\quad \hat y_1=(b_0+b_2)+(b_1+b_3)x.$

The prediction difference is $\hat y_1-\hat y_0=b_2+b_3x$. If $b_3\ne0$, the lines meet at $x=-b_2/b_3$.

Use it. Problem 6 gives $b_0=35$, $b_1=5$, $b_2=-15$, and $b_3=5$. The lines are $35+5x$ and $20+10x$; their crossing is $x=-(-15)/5=3$.

Adding a term such as $b_4x^2g$ leaves the $g=0$ group linear and permits curvature for $g=1$. When both fits minimize ordinary unpenalized RSS on the same training data, the extended model includes the old model at $b_4=0$, so:

$\mathrm{RSS}_{\text{extended}}\le\mathrm{RSS}_{\text{original}}.$

These fully interacted models allow independent group fits. With unconstrained coefficients, adding curvature only for $g=1$ leaves the minimum $g=0$ training RSS unchanged. If the $g=0$ line is identifiable (at least two distinct training $x$ values), its predictions and its RSS on the same test points also stay unchanged. Total test RSS has no guaranteed ordering.

Shifted noise · A nonzero mean changes the intercept

Suppose the model is $Y=\beta_0+\beta_1X+\varepsilon$, with noise independent of $X$ and $\varepsilon\sim N(m,\sigma^2)$. Here $m$ is its known mean and $\sigma^2$ its variance. Then:

$E[Y\mid X=x]=(\beta_0+m)+\beta_1x.$

An ordinary fitted intercept $\hat\alpha_0$ estimates the mean-response intercept $\alpha_0=\beta_0+m$. Recover the original model’s intercept by subtracting the known noise mean:

$\hat\beta_0=\hat\alpha_0-m.$

If the interval for $\alpha_0$ is $[L,U]$, the interval for $\beta_0$ is $[L-m,U-m]$. Its width is unchanged because $m$ is a known constant. The slope estimate and its uncertainty are unchanged for the same observations.

Use it. Problem 7’s printed data give $\hat\alpha_0=0$. When the known noise mean is $m=1$, the original intercept estimate is $0-1=-1$. Its 95% interval is $-1\pm1.96w/\sqrt5$, where $w$ is the supplied noise standard deviation.

If $m$ is unknown, these observations identify the sum $\beta_0+m$, not its two components separately.