Midterm practice

The price is the response, not a fourth predictor. For the boundary, the denominator $x^2+1/2$ is positive, so multiplying an inequality by it keeps the inequality’s direction.

1.1 — Identify the task. Price is numerical, so this is regression. There are $n=1000$ houses and $p=3$ predictors: bedrooms, bathrooms, school scores. Understanding which factors are associated with price is an inference goal; it does not by itself establish causation.

1.2 — Find when class 1 is more probable. Substitute the supplied probability into $p_1(x)\ge1/2$:

Thus $x^2\ge3/2$. On the given domain $|x|\ge1/\sqrt2$, the classifier is

Check: at $x=1$, $p_1=(1-0.5)/(1+0.5)=1/3$, so predict 0. At $x=2$, $p_1=(4-0.5)/(4+0.5)=7/9$, so predict 1. Do not extend the rule to the excluded middle interval: the supplied expression is not a valid probability there.

1.3 — Name what the error means. If “positive” means disease, a false negative misses someone who has the disease; a false positive flags someone who does not. True under the usual screening assumption that missing a serious disease is costlier than an unnecessary follow-up. It is not a universal fact about all medical decisions; the consequence of each error determines the cost.

When those costs differ, the usual $1/2$ threshold need not minimize cost. With false-positive cost $c_{FP}$ and false-negative cost $c_{FN}$, predict 1 when

A larger cost for missed positives lowers the threshold.

Classification; $n=200$ emails and $p=4$ predictors. Predict 0 because $0.4<0.5$. The supplied spam label is the response, not a fifth predictor.

Find $\bar x$, $\bar y$, $S_{xx}$, and $S_{xy}$. Reuse them for both the fitted line and its standard errors. For a hypothesis test, compare the distance from the estimate to the proposed value with the interval’s half-width.

2.1 · Find the fitted line

Means. There are $n=5$ points. Their $x$ values sum to 5 and their $y$ values sum to 0, so the mean rule gives $\bar x=5/5=1$ and $\bar y=0/5=0$.

Spread and joint variation. Subtract $\bar x=1$ from each $x$, giving $(-2,-1,0,1,2)$. Since $\bar y=0$, the centered $y$ values are $(-3,-2,0,2,3)$. Squaring the centered $x$ values and multiplying the matching centered pairs gives:

$S_{xx}=4+1+0+1+4=10$

$S_{xy}=6+2+0+2+6=16.$

Slope. Divide joint variation $S_{xy}=16$ by horizontal spread $S_{xx}=10$:

$\hat\beta_1=\frac{16}{10}=1.6.$

Intercept. The fitted line passes through $(\bar x,\bar y)=(1,0)$, so:

$\hat\beta_0=0-(1.6)(1)=-1.6.$

The answer is $\hat y=-1.6+1.6x$. Increasing $x$ by 1 increases the fitted response by 1.6.

2.2 · Measure the fit

Residuals. Substitute each supplied $x$ into $\hat y=-1.6+1.6x$, then calculate $e=y-\hat y$:

RSS. Square and add the residuals $(0.2,-0.4,0,0.4,-0.2)$:

$\mathrm{RSS}=0.04+0.16+0+0.16+0.04=0.4.$

RSE. We fitted two coefficients using $n=5$ points, leaving $5-2=3$ residual degrees of freedom. With $\mathrm{RSS}=0.4$:

$\mathrm{RSE}=\sqrt{\frac{0.4}{5-2}}\approx0.3651.$

$R^2$. With $\bar y=0$ and observed $y=(-3,-2,0,2,3)$, the baseline squared error is:

$\mathrm{TSS}=9+4+0+4+9=26.$

Using $\mathrm{RSS}=0.4$ and $\mathrm{TSS}=26$:

$R^2=1-\frac{0.4}{26}=\frac{64}{65}\approx0.9846.$

The line removes about 98.46% of the squared error made by predicting the mean for these training points.

2.3 · Find the 95% confidence intervals

Slope uncertainty. The question supplies $\sigma=\sqrt8\,s$; the data give $S_{xx}=10$. Substitute into the slope standard-error rule:

$\mathrm{SE}(\hat\beta_1)=\frac{\sqrt8\,s}{\sqrt{10}}=\sqrt{\frac45}\,s.$

With slope estimate $\hat\beta_1=1.6$, its interval is:

$\beta_1:\ 1.6\pm1.96\sqrt{\frac45}\,s$

or approximately $1.6\pm1.7531s$.

Intercept uncertainty. Use $\sigma=\sqrt8\,s$, $n=5$, $\bar x=1$, and $S_{xx}=10$:

$\mathrm{SE}(\hat\beta_0)=\sqrt8\,s\sqrt{\frac15+\frac{1^2}{10}}$

$=\sqrt{\frac{12}{5}}\,s.$

With intercept estimate $\hat\beta_0=-1.6$, its interval is:

$\beta_0:\ -1.6\pm1.96\sqrt{\frac{12}{5}}\,s$

or approximately $-1.6\pm3.0364s$. Larger assumed noise makes both intervals wider.

2.4 · When do we reject a slope of 4?

The fitted slope is 1.6 and the proposed slope is 4, so their distance is $|1.6-4|=2.4$. Reject when that distance exceeds the 95% half-width $1.96\sqrt{4/5}\,s$:

$2.4>1.96\sqrt{\frac45}\,s.$

Since $s>0$, this gives:

$0<s<\frac{2.4}{1.96\sqrt{4/5}}$

The upper threshold is approximately 1.3690. With enough noise, the observed slope is no longer sufficiently far from 4 to reject it.

2.5 · When can we not reject an intercept of 2?

The fitted intercept is −1.6 and the proposed intercept is 2, so their distance is $|-1.6-2|=3.6$. We fail to reject when this distance is at most the half-width $1.96\sqrt{12/5}\,s$:

$3.6\le1.96\sqrt{\frac{12}{5}}\,s.$

Solving for $s$ gives:

$s\ge\frac{3.6}{1.96\sqrt{12/5}}\approx1.1856.$

Failing to reject means the data are compatible with the proposed value at this noise level. It does not establish that the intercept equals 2. We use strict rejection outside the interval; a value exactly at an endpoint is retained.

The new mean is $\bar y=5$, while $\bar x=1$, $S_{xx}=10$, and $S_{xy}=16$ stay unchanged because centered $y$ values stay unchanged. Thus the slope remains $16/10=1.6$, and the intercept becomes $5-(1.6)(1)=3.4$. Both observed and fitted responses rise by 5, so every residual stays unchanged and RSS remains 0.4.

At the origin $(0,0)$, the score is $2>0$. That means $P(Y=0)>1/2$, so this side predicts 0. Draw $x_2=x_1+2$ through $(-2,0)$ and $(0,2)$, then count each color on each side. None of the drawn points lies on the line.

3.1 — Set the two class probabilities equal. Because $\sigma(z)=1/2$ exactly when $z=0$, substitute $z=2+x_1-x_2$:

Draw the line through $(-2,0)$ and $(0,2)$. Above the line, $2+x_1-x_2<0$, so class 0 has probability below $1/2$ and we predict 1. Below the line, predict 0. Use 1 at exact ties, as elsewhere in this module.

This sketch shows the decision rule. Use the original drawing above to count the observations.

3.2 — Count by true class first. Of the 7 red positives, 4 lie above the line ($TP$) and 3 below it ($FN$). Of the 6 blue negatives, 2 lie above the line ($FP$) and 4 below it ($TN$). The hand-drawn coordinates need not be exact to establish which side each point occupies.

So $FN=3$, $FP=2$. Check: $4+3+2+4=13$ points, with $3+2=5$ mistakes.

3.3 — Flip the predictions. $C_{NB}(x)=1-C_B(x)$ changes each 1 to 0 and each 0 to 1. The actual labels stay fixed, so swap the two prediction columns:

Now $FN=4$, $FP=4$. Check: the original 8 correct predictions become 8 mistakes.

3.4 — Predict 1 for everyone. All 7 positives are found: $TP=7$, $FN=0$. All 6 negatives are falsely flagged: $FP=6$, $TN=0$. Substitute those counts into the rate formulas:

Zero missed positives does not mean a useful classifier: this one also flags every negative.

There are $FN=10-8=2$ missed positives, so $\mathrm{FNR}=2/10=0.2$. There are $FP=3$ falsely flagged negatives, so $\mathrm{FPR}=3/15=0.2$. Dividing either count by all 25 observations would answer a different question.

For rates, begin with “among actual spam” or “among actual non-spam.” For noise, $N(1,2)$ has mean 1 and variance 2. For independence, try a symmetric $X$ and the variable $X^2$. For model selection, training error and new-data error answer different questions.

4a — Choose the more probable class. Substitute $p_1(x)=1/(1+x^2)$ into $p_1(x)\ge1/2$. Since $1+x^2>0$,

Therefore predict 1 on $[-1,1]$, and 0 outside. The endpoints are ties, assigned 1 here. Check: $x=0$ gives probability 1; $x=2$ gives $1/5$.

4b — Say who is in the denominator. Let positive mean spam.

Every actual spam email is either caught or missed, so $\mathrm{TPR}=1-\mathrm{FNR}$. FPR is not the fraction of flagged emails that are innocent: that would use a different denominator.

4c — A common cause can move both variables. Hotter weather can increase both ice-cream sales and swimming activity. Sales and swimming can be correlated without sales causing swimming. Correlation alone does not establish causation. A shared cause, reverse causation, or other mechanisms can create the association.

4d — Independence is stronger than zero correlation. With finite variances, independence gives $E[UV]=E[U]E[V]$, so covariance is zero. Correlation is then zero if both variances are nonzero.

The reverse fails. As a counterexample, let $X$ be equally likely to be $-1,0,1$ and let $Z=X^2$. These are invented values to demonstrate the claim. Since $X$ determines $Z$, the variables are dependent. But symmetry gives

Using $\operatorname{Cov}(X,Z)=E[XZ]-E[X]E[Z]$ gives $0-0(2/3)=0$. Both variances are positive, so the correlation is zero despite dependence.

4e — Average the noise; keep its mean. Use the usual independent-noise assumption (the marginal distribution alone would not specify $E[\varepsilon\mid X=x]$). The supplied $N(1,2)$ has mean 1 and variance 2. With $h(x)=x+x^2$,

After predicting this average, the leftover error is $\varepsilon-1$. Its mean squared size is

Using $E[\varepsilon^2]=2+1^2=3$ would incorrectly count the predictable mean as unavoidable error.

4f — Compare new-data error at each flexibility. Fit each candidate using training data, calculate its $MSE_{Te}$ on held-out observations, and select the flexibility with the lowest estimated new-data MSE. Training MSE alone can reward fitting noise.

In a practical workflow, use a validation set or cross-validation for this choice, then use a separate final test set once. Repeatedly using a “test” set to select models makes it a validation set; its minimum error is no longer an untouched final assessment.

4g — False: the issue is high variance. A very flexible model can chase accidental patterns in its training sample. On a different sample, the fitted curve may change substantially: high variance. A model too rigid to capture the pattern typically has high bias and underfits. Increasing flexibility often reduces bias while increasing variance.

4h — Choose the class with the highest conditional probability. At input $x$, predicting class $k$ is correct with probability $P(Y=k\mid X=x)$. Making that probability as large as possible makes the chance of error as small as possible.

With equal error costs, this minimizes expected $FP+FN$ on fresh observations. It does not guarantee the fewest realized mistakes on every finite sample; random outcomes can differ from expectations. It also need not minimize $FN$ alone. Predicting 1 for everyone makes $FN=0$, but can create many false positives, as in Problem 3.

The conditional mean is $f(x)=3x-2$. At $x=4$, predict $3(4)-2=10$. The irreducible MSE is the noise variance, 9, not the standard deviation 3 and not $E[\varepsilon^2]=13$.

It does not satisfy part (a), because the order differs from H, H, T. Part (b) accepts every arrangement with the requested counts, so you need a counting factor.

5a — Multiply along the specified order. Independence lets us multiply the probabilities of H, H, and T, respectively $p$, $p$, and $1-p$:

There is no factor of 3 because only this particular order qualifies.

5b — Count qualifying orders, then multiply. One sequence with 20 heads and 10 tails has probability $p^{20}(1-p)^{10}$. The number of distinct arrangements is

Those arrangements are mutually exclusive; only one order occurs. Add their equal probabilities:

Keep $p$ as a variable: the question asks for probability as a function of $p$, not an estimate of the coin’s bias. As a check, if we assume $p=1/2$, the result is $30{,}045{,}015/2^{30}\approx0.02798$.

One order: $0.6^2(0.4)=0.144$. Any order: there are $\binom32=3$ qualifying sequences (HHT, HTH, THH), so $3(0.144)=0.432$.

Set $g=0$ and then $g=1$. For part 6.d, ask whether the shared-looking coefficients actually constrain the two groups to share a line. Rewrite them as one intercept and slope for each group.

6.a · Plot the two regression functions

Male group. Substitute $g=0$ into the supplied model; both terms containing $g$ disappear:

$\hat y_M=35+5x.$

Female group. Substitute $g=1$, combine the intercept terms $35-15$, and combine the slope terms $5x+5x$:

$\hat y_F=35+5x-15+5x$

$\hat y_F=20+10x.$

Use these calculated points to draw the two straight lines:

At GPA 3, both models predict $50,000. These are fitted group averages, not a causal claim about gender.

6.b · Where is the Male prediction higher?

Compare the fitted functions $35+5x$ and $20+10x$ by subtraction:

$\hat y_M-\hat y_F=15-5x.$

A higher Male prediction means a positive difference: $15-5x>0$, or $x<3$. Within the requested GPA range, the answer is $0\le x<3$. The groups tie at 3; the Female prediction is higher for $3<x\le4$.

6.c · Compare total training RSS

The extended fit adds $\tilde\beta_4x^2g$. Setting $\tilde\beta_4=0$ reproduces every fit available to the old model. Minimizing over this larger choice of fits therefore gives:

$\mathrm{RSS}_{\text{new, train}}\le\mathrm{RSS}_{\text{old, train}}.$

The new RSS is no larger. A strict decrease is not guaranteed; the extra term might provide no improvement. Although the model is quadratic in GPA for one group, it is still linear in its fitted coefficients, so ordinary least squares applies.

6.d · Compare Male training RSS

For ordinary unconstrained least squares, the Male training RSS is the same in both fits. Merely observing that $x^2g=0$ for Males is not enough: the other coefficients are also refitted. The reason is that this model can fit the groups independently.

Define separate group coefficients:

$a_M=b_0,\qquad c_M=b_1$

$a_F=b_0+b_2,\qquad c_F=b_1+b_3.$

Any two intercepts and slopes can be represented: choose $b_2=a_F-a_M$ and $b_3=c_F-c_M$. Thus the original fit independently minimizes a Male line’s RSS and a Female line’s RSS. The extended fit independently minimizes a Male line’s RSS and a Female quadratic’s RSS.

The Male task is identical in both cases: minimize $\sum_{i\in M}(y_i-a_M-c_Mx_i)^2$. Its optimal training RSS cannot change. With at least two distinct Male GPA values, its fitted intercept and slope are also identical. This conclusion assumes unpenalized least squares with no constraints linking the group coefficients.

6.e · What changes on test data?

All groups combined: there is no guaranteed ordering of test RSS. The Female quadratic may capture useful curvature or fit training noise, so total test RSS may fall, rise, or stay equal.

Male points only: the two fits make the same Male predictions at every GPA when the Male line is identifiable (at least two distinct training GPA values). On the same Male test points, their test RSS is therefore equal too. Without an identifiable Male line, off-training predictions need not be unique, even though minimum Male training RSS is equal.

Setting $g=0$ gives Male salary $35+5x$; setting $g=1$ gives Female salary $20+8x$. Their difference is $15-3x$. Setting it to zero gives $x=15/3=5$, outside $[0,4]$. The Male prediction is higher throughout the requested range. At GPA 4, the predictions are $35+5(4)=55$ and $20+8(4)=52$, in thousands of dollars.

Both coordinate means are zero. This makes the fitted mean-response intercept zero and removes the $\bar x^2/S_{xx}$ term from its variance. In part 7.d, distinguish the fitted mean-response line from the structural parameter $\beta_0$ in the model statement.

7.a · Fit the coefficients

Means. The five $x$ values sum to 0, and the five $y$ values sum to 0. With $n=5$, $\bar x=0/5=0$ and $\bar y=0/5=0$.

Spread. With $\bar x=0$, square the supplied $x=(-2,-1,0,1,2)$ values:

$S_{xx}=4+1+0+1+4=10.$

Joint variation. Because both means are zero, multiply each supplied $(x,y)$ pair and add:

$S_{xy}=7+3+0+3+7=20.$

Slope and intercept. Substitute $S_{xy}=20$, $S_{xx}=10$, $\bar x=0$, and $\bar y=0$:

$\hat\beta_1=20/10=2$

$\hat\beta_0=0-(2)(0)=0.$

The fitted line is $\hat y=2x$. Here the original noise mean is zero, so the fitted intercept estimates $\beta_0$ directly.

7.b · Find the confidence intervals

Slope. The supplied noise standard deviation is $w$, and $S_{xx}=10$, so $\mathrm{SE}(\hat\beta_1)=w/\sqrt{10}$. With fitted slope 2:

$\beta_1:\ 2\pm1.96\frac{w}{\sqrt{10}}$

or approximately $2\pm0.6198w$.

Intercept. Use $n=5$, $\bar x=0$, $S_{xx}=10$, and noise standard deviation $w$:

$\mathrm{SE}(\hat\beta_0)=w\sqrt{\frac15+\frac{0^2}{10}}=\frac{w}{\sqrt5}.$

With fitted intercept 0, its interval is:

$\beta_0:\ 0\pm1.96\frac{w}{\sqrt5}$

or approximately $0\pm0.8765w$. These use the supplied noise variance, so the normal multiplier is 1.96.

7.c · When do we reject a zero slope?

The fitted slope 2 is a distance $|2-0|=2$ from the null slope 0. Reject at the two-sided 5% level when this exceeds the half-width $1.96w/\sqrt{10}$:

$2>1.96\frac{w}{\sqrt{10}}.$

Solving for positive $w$ gives:

$0<w<\frac{2\sqrt{10}}{1.96}\approx3.2268.$

At larger noise levels, zero falls inside the slope’s interval. A value exactly at the endpoint is retained under our strict rejection rule.

7.d · What if the noise mean is 1?

The fitted mean-response line stays the same. For the same printed observations, ordinary least squares still returns intercept $\hat\alpha_0=0$ and slope $\hat\beta_1=2$. Changing an assumption does not change the data’s least-squares line.

The original intercept’s interpretation changes. With the supplied new noise mean $m=1$:

$E[Y\mid X=x]=(\beta_0+1)+\beta_1x.$

The fitted intercept 0 now estimates $\beta_0+1$. Subtract the known mean to estimate $\beta_0$:

$\hat\beta_0=\hat\alpha_0-1=0-1=-1.$

The variance is still $w^2$. Therefore the 95% interval for the original intercept shifts down by 1 and keeps its half-width:

$\beta_0:\ -1\pm1.96\frac{w}{\sqrt5}.$

The slope estimate remains 2, its interval remains $2\pm1.96w/\sqrt{10}$, and the rejection condition remains $0<w<3.2268$ (approximately). The predicted average response is still $\hat y=(-1)+2x+1=2x$.

The data still give mean-response intercept $\hat\alpha_0=0$. Subtracting the known noise mean $m=-2$ gives $\hat\beta_0=0-(-2)=2$. With $n=5$, $\bar x=0$, and $w=1$, its standard error is $1/\sqrt5$. The interval is $2\pm1.96/\sqrt5$, approximately $[1.1235,2.8765]$. Its center changes; its uncertainty comes from the variance, which has not changed.

Nothing changes the coin. Whether the observation is actually positive or negative, the probability of predicting 1 is still $p$.

8a — Calculate each coordinate. The coin is independent of the actual label. Among actual negatives it predicts 1 with probability $p$, so $\mathrm{FPR}=p$. Among actual positives it also predicts 1 with probability $p$, so $\mathrm{TPR}=p$.

Thus the curve is $\mathrm{TPR}=\mathrm{FPR}$, a diagonal from $(0,0)$ to $(1,1)$. For example, $p=0.2$ gives $(0.2,0.2)$; $p=0.8$ gives $(0.8,0.8)$.

8b — Find the area of the triangle. The area below that diagonal has base 1 and height 1:

8c — No information means no systematic ranking advantage or disadvantage. If a score or prediction is independent of the true label, positives and negatives have the same score distribution. Every threshold therefore produces equal TPR and FPR. You cannot systematically achieve a population AUC above or below $1/2$ without information related to the label. Flipping an independent coin’s outputs still gives an independent coin.

A measured AUC on a finite random sample can be above or below $1/2$ by chance. Also, knowing that one class is more common can help accuracy through majority guessing, but supplies no way to rank individual positives ahead of negatives; it does not improve AUC.

With $p=0.3$, FPR and TPR are both 0.3, and FNR is $1-0.3=0.7$. Raising $p$ to 0.8 moves along the same diagonal. The family still has AUC $1/2$.