Module 6 · Formula sheet | principles.fyi

Bayes and joint densities

Weight each class by how common it is and how well it fits the input. Let $Y=k$ name a class, $x$ be an observed continuous input, $\pi_k=P(Y=k)$ its prior, and $f_k(x)$ the density of $X$ within that class. There are $K$ classes.

The joint density in $x$ and discrete label $k$ is

g(x,k)=\pi_kf_k(x).

Its area over all $x$ is $\pi_k$ . A conditional density $f_k$ has area 1. Density is curve height; the probability of a continuous input interval is its area, and $P(X=x)=0$ at an exact point.

The posterior, or class probability after seeing $x$ , is

P(Y=k\mid X=x)= \frac{\pi_kf_k(x)}{\sum_{\ell=1}^{K}\pi_\ell f_\ell(x)}.

$\ell$ runs through all class labels. For equal mistake costs, choose

C(x)=\arg\max_k\{\pi_kf_k(x)\}.

$C(x)$ is the predicted label; arg max returns the label with the largest weight. The denominator is the same for every candidate class.

Invented numbers: priors $0.8,0.2$ and densities $0.1,0.3$ give weights $0.08,0.06$ . Class 1’s posterior is $0.06/(0.08+0.06)\approx0.429$ ; predict class 0.

Q3: labels $y\in\{-1,+1\}$ , means $3y$ , shared variance 1, and priors $1/2$ give

g(x,y)=\frac{1}{2\sqrt{2\pi}} \exp\!\left[-\frac{(x-3y)^2}{2}\right].

Univariate LDA

Model one bell curve per class with a shared spread. For input $X$ and class $Y=k$ :

X\mid Y=k\sim\mathcal N(\mu_k,\sigma^2),

f_k(x)=\frac{1}{\sigma\sqrt{2\pi}} \exp\!\left[-\frac{(x-\mu_k)^2}{2\sigma^2}\right].

$\mu_k$ is the class mean; $\sigma^2>0$ is the common variance; $\sigma$ is the standard deviation. Here $\pi$ is the mathematical constant and $\pi_k=P(Y=k)>0$ is the class prior. $\exp(a)=e^a$ .

Instead of comparing weighted densities $\pi_kf_k(x)$ , compare linear discriminant scores:

\delta_k(x)=\frac{\mu_kx}{\sigma^2} -\frac{\mu_k^2}{2\sigma^2}+\log\pi_k.

$\log$ is the natural logarithm. Predict the label with the largest score. A discriminant score is not itself a probability.

For classes 0 and 1 with $\mu_1>\mu_0$ , equal mistake costs give input boundary

\begin{aligned} \gamma&=\frac{\mu_0+\mu_1}{2}\\ &\quad+\frac{\sigma^2}{\mu_1-\mu_0} \log\frac{\pi_0}{\pi_1}. \end{aligned}

Predict 1 for $x\geq\gamma$ , otherwise 0. Equal priors give $\log(1)=0$ , hence the midpoint. If the means are reversed, reverse the direction of the threshold rule. If they coincide, identical densities leave only the priors to compare.

Q7: means $-1,+1$ , variance $1/4$ , and priors $1/2,1/2$ give $\gamma=(-1+1)/2=0$ . Q3: means $-3,+3$ and equal priors also give $\gamma=0$ , using labels $-1,+1$ .

Estimating LDA from data

Count classes, average within classes, then pool the within-class spread. Let training pair $(x_i,y_i)$ contain input $x_i$ and true label $y_i$ . There are $N$ observations and $K$ classes; $N_k$ is class $k$ ‘s count. Hats mean estimates.

\widehat\pi_k=\frac{N_k}{N}, \qquad \widehat\mu_k=\frac1{N_k}\sum_{i:y_i=k}x_i.

The shared estimated variance is

\widehat\sigma^2= \frac{\sum_{k=1}^{K}\sum_{i:y_i=k} (x_i-\widehat\mu_k)^2}{N-K}.

Each difference uses its own class’s mean. If the class sample variances $s_k^2$ are given, the equivalent rule is

\widehat\sigma^2= \frac{\sum_{k=1}^{K}(N_k-1)s_k^2}{N-K},

where $s_k^2=\sum_{i:y_i=k}(x_i-\widehat\mu_k)^2/(N_k-1)$ when $N_k\geq2$ . The denominator $N-K$ accounts for estimating one mean per class. The pooled-deviations rule needs each class represented, $N>K$ , and a positive result to use as an LDA variance.

Invented data: class 0 inputs $-2,0$ ; class 1 inputs $2,4$ . Then $N=4$ , $K=2$ , each $N_k=2$ ; priors are $2/4=0.5$ . Means are $(-2+0)/2=-1$ and $(2+4)/2=3$ . Squared within-class deviations sum to $1+1+1+1=4$ , so pooled variance is $4/(4-2)=2$ .

Insert these estimates into the LDA score to classify new inputs.

Multivariate LDA

Several inputs make a cloud rather than a one-dimensional bell curve. Let $\mathbf x$ contain $p$ input values, $\boldsymbol\mu_k$ contain the $p$ class means, and $\Sigma$ be the shared nonsingular $p\times p$ covariance matrix. Covariance records input spreads and how inputs move together.

Define the adjusted squared distance

\begin{aligned} \mathbf u_k&=\mathbf x-\boldsymbol\mu_k,\\ d_k^2&=\mathbf u_k^T\Sigma^{-1}\mathbf u_k. \end{aligned}

Then the class density is

f_k(\mathbf x)= \frac{\exp(-d_k^2/2)}{(2\pi)^{p/2}|\Sigma|^{1/2}}.

$\mathbf u_k$ is the offset from the class center; $T$ denotes transpose; $\Sigma^{-1}$ adjusts for spread and correlation; $|\Sigma|$ is the determinant; $\pi$ is the mathematical constant. With class priors $\pi_k=P(Y=k)$ , predict the class with largest $\pi_kf_k(\mathbf x)$ .

For $p=1$ , $\Sigma=\sigma^2$ and $d_k^2=(x-\mu_k)^2/\sigma^2$ , giving the usual univariate normal density. Shared covariance gives linear class boundaries: lines for two inputs, flat surfaces in higher dimensions.

Confusion matrix rates

Name the actual class before choosing the denominator. Positive means class 1; negative means class 0. These symbols are counts:

Prediction	Actual 0	Actual 1
Predict 0	True negative (TN)	False negative (FN)
Predict 1	False positive (FP)	True positive (TP)

Among actual negatives:

\begin{aligned} FPR&=\frac{FP}{FP+TN},\\ TNR&=\frac{TN}{FP+TN}=1-FPR. \end{aligned}

Among actual positives:

\begin{aligned} FNR&=\frac{FN}{FN+TP},\\ TPR&=\frac{TP}{FN+TP}=1-FNR. \end{aligned}

The denominator must be nonzero. Overall, with $N=TP+TN+FP+FN$ :

\text{Error rate}=\frac{FP+FN}{N}.

Slide training counts: $TN=9644$ , $FN=252$ , $FP=23$ , $TP=81$ . Hence $FPR=23/9667\approx0.238\%$ , $FNR=252/333\approx75.68\%$ , $TPR=81/333\approx24.32\%$ , and training error $=(23+252)/10000=2.75\%$ . These describe the training data; test performance needs new observations.

Q4, spam is positive: FPR is legitimate email incorrectly marked spam, divided by all actual legitimate email. FNR is missed spam divided by all actual spam. TPR is caught spam divided by all actual spam.

Count mistakes with indicators

For observation $i$ , let $y_i$ be its true label and $C(x_i)$ its predicted label. An indicator $\mathbf1\{A\}$ equals 1 if statement $A$ is true, otherwise 0:

FP=\sum_{i=1}^{N} \mathbf1\{C(x_i)=1\}\mathbf1\{y_i=0\},

FN=\sum_{i=1}^{N} \mathbf1\{C(x_i)=0\}\mathbf1\{y_i=1\}.

Both switches must be on to count a row. The slides’ equality switch $\delta(a)=1$ for $a=0$ , otherwise 0, serves the same purpose.

Gaussian tail errors

Standardize the boundary, then take the area on the wrong side. Suppose classes 0 and 1 have normal means $\mu_0,\mu_1$ , shared standard deviation $\sigma>0$ , and the rule predicts 1 for $x\geq\gamma$ .

The standard-normal cumulative distribution function $\Phi(z)=P(Z\leq z)$ gives the area to the left of $z$ for $Z\sim\mathcal N(0,1)$ . The standardized boundary for class $k$ is

z_k=\frac{\gamma-\mu_k}{\sigma}.

For class density $f_k(x)$ , conditional error rates are

\begin{aligned} FPR&=\int_\gamma^\infty f_0(x)\,dx\\ &=1-\Phi(z_0), \end{aligned}

\begin{aligned} FNR&=\int_{-\infty}^\gamma f_1(x)\,dx\\ &=\Phi(z_1). \end{aligned}

Then $TPR=1-FNR$ .

Each conditional density has total area 1. An integral adds curve area, giving probability.

Q3: use negative-class mean $-3$ , $\sigma=1$ , and boundary $0$ . Then $z=[0-(-3)]/1=3$ . The 99.7% empirical rule gives one tail $\approx(1-0.997)/2=0.0015$ . Exact normal area is $1-\Phi(3)\approx0.001350$ .

Q7: use means $-1,+1$ , $\sigma=1/2$ , and boundary $0$ . Then $z_0=[0-(-1)]/(1/2)=2$ and $z_1=(0-1)/(1/2)=-2$ . The practice approximation gives

\begin{aligned} FPR\approx FNR&\approx\frac{1-0.954}{2}\\ &=0.023, \end{aligned}

TPR\approx1-0.023=0.977.

Exact Gaussian values are $FPR=FNR\approx0.022750$ and $TPR\approx0.977250$ . Use either exact values or the stated empirical approximation consistently.

Population error and expected counts

Conditional rates have different denominators; use priors to combine them. Let $\pi_0=P(Y=0)$ and $\pi_1=P(Y=1)$ be actual class frequencies, with $\pi_0+\pi_1=1$ . Let $FPR=P(\widehat Y=1\mid Y=0)$ and $FNR=P(\widehat Y=0\mid Y=1)$ , where $\widehat Y$ is the predicted label.

P(\text{false positive})=\pi_0FPR,

P(\text{false negative})=\pi_1FNR,

\boxed{\text{Error}=\pi_0FPR+\pi_1FNR.}

For a batch of $N$ observations drawn from this population, $E$ denotes an expected count:

\begin{aligned} E[FP]&=N\pi_0FPR,\\ E[FN]&=N\pi_1FNR. \end{aligned}

Q7: equal priors $0.5,0.5$ and the approximate rates $0.023,0.023$ give error $0.5(0.023)+0.5(0.023)=0.023=2.3\%$ . An invented batch of $N=1000$ has expected FP $=1000(0.5)(0.023)=11.5$ and the same expected FN, or 23 expected mistakes total.

Expected counts can be fractional; observed counts cannot. For a dataset, using its observed class proportions and observed conditional rates recovers $(FP+FN)/N$ exactly.

Shifted Gaussian inputs

Adding a constant moves the center without changing the spread. Suppose

\begin{aligned} X&=aY+\varepsilon,\\ \varepsilon\mid Y=k&\sim\mathcal N(m,s^2). \end{aligned}

$Y=k$ is a class label; $a$ is a supplied multiplier; $m$ is the noise mean and $s^2$ its variance. The same noise distribution is assumed in each class. Substituting $Y=k$ gives

X\mid Y=k\sim\mathcal N(ak+m,s^2).

With $s>0$ , its density is

f_k(x)=\frac1{s\sqrt{2\pi}} \exp\!\left[-\frac{(x-ak-m)^2}{2s^2}\right].

Q7 supplies $a=2$ , $m=-1$ , $s^2=1/4$ , and $k\in\{0,1\}$ . Thus means are $2(0)-1=-1$ and $2(1)-1=1$ , with standard deviation $s=\sqrt{1/4}=1/2$ in both classes:

f_k(x)=\sqrt{\frac2\pi}\exp[-2(x+1-2k)^2].

Equal transmitted frequencies give $\pi_0=\pi_1=1/2$ . The Gaussian’s second parameter is variance, so do not use $1/4$ as the standard deviation.

Thresholds and ROC

The threshold chooses the balance of misses and false alarms. Let $p_1(x)=P(Y=1\mid X=x)$ be the posterior for class 1 and $\tau$ a probability threshold:

C_\tau(x)=\begin{cases} 1,&p_1(x)\geq\tau,\\ 0,&p_1(x)<\tau. \end{cases}

Equal costs for false positives and false negatives give $\tau=0.5$ . Lowering $\tau$ can increase TPR and FPR while decreasing FNR; raising it reverses those changes. Counts may stay unchanged between observed scores.

For univariate LDA with class means $\mu_1>\mu_0$ , common variance $\sigma^2>0$ , priors $\pi_0,\pi_1>0$ , and $0<\tau<1$ , the equivalent input boundary is

\begin{aligned} \gamma_\tau&=\frac{\mu_0+\mu_1}{2}\\ &\quad+\frac{\sigma^2}{\mu_1-\mu_0} \left[\log\frac{\pi_0}{\pi_1} +\log\frac{\tau}{1-\tau}\right]. \end{aligned}

Predict 1 for $x\geq\gamma_\tau$ . $\tau$ is a probability; $\gamma_\tau$ has the input’s units. $\log$ is the natural logarithm.

Q7 model, invented threshold: means $-1,+1$ , variance $1/4$ , equal priors, and $\tau=0.2$ give

\begin{aligned} \gamma_{0.2} &=0+\frac{1/4}{2}\log\frac{0.2}{0.8}\\ &=\frac18\log(0.25)\approx-0.1733. \end{aligned}

Moving left from the usual boundary 0 predicts more positives, catching more actual ones and also more actual zeros.

An ROC curve plots $FPR$ horizontally and $TPR=1-FNR$ vertically while varying the threshold. Here FPR is the fraction of actual negatives flagged positive; TPR is the fraction of actual positives caught. Each point is one threshold. The desirable corner is $(0,1)$ .

AUC is the area under the ROC curve: 1 for perfect separation and 0.5 for a diagonal curve. It summarizes separation across thresholds; costs and acceptable error rates still determine which threshold to use.