Module 5 · Formula sheet

Back to the Module 5 guide.

Class probabilities and Bayes classification

Choose the most likely label when all mistakes count equally. For input $x$, the conditional class probability is

$X$ is the input, $Y$ the label, $k$ a possible label, and $\mid$ means “given.” For $K$ classes, their probabilities sum to 1. With two labels, 0 and 1, $p_0(x)=1-p_1(x)$.

The Bayes optimal classifier uses the true probabilities:

“Arg max” returns the label with the largest probability. With two labels, choose 1 when $p_1(x)\geq0.5$ and 0 otherwise; either label is equally good at a tie, which we assign to 1.

Invented example: if $p_1(x)=0.8$, predicting 1 has error probability $1-0.8=0.2$; predicting 0 has error probability 0.8. The Bayes rule chooses 1. With estimated probabilities, the same decision rule gives an estimated classifier, not necessarily the true optimum.

Local averaging

Estimate a probability by counting nearby labels. For binary training labels $y_i\in\{0,1\}$, choose a neighborhood $\mathcal N(x)$ containing $m>0$ observations near input $x$:

The index $i$ selects a training row, $m$ is the number of selected rows, and a hat marks an estimate. Since labels are 0 or 1, the sum counts class-1 observations. Choose class 1 when $\widehat p_1(x)\geq0.5$.

Invented neighborhood: $1,1,0,1,0$ gives $\widehat p_1=3/5=0.6$ and $\widehat p_0=0.4$, so predict 1.

In many dimensions, small neighborhoods can be nearly empty and large ones can include dissimilar inputs: the curse of dimensionality.

Horizontal shifts

Work with the expression inside the function. If $R(u)=1$ for $a\leq u\leq b$ and 0 otherwise, then

$a,b$ are the original interval endpoints; $c$ is the amount added to the input. For $c>0$, $R(x+c)$ shifts left by $c$; $R(x-c)$ shifts right by $c$. The height is unchanged. Included endpoints stay included.

Shared Q1: $a=-2$, $b=2$, $c=1$. Thus $R(x+1)=1$ on $[-2-1,2-1]=[-3,1]$ and 0 outside. At $x=-3$, the inside is $-3+1=-2$, so the value is 1.

Sigmoid identities

The sigmoid turns a real score into a probability between 0 and 1. For real number $z$:

$S$ names the sigmoid; $e\approx2.718$ is the exponential base. Multiply numerator and denominator of the first form by $e^z$ to get the second, using $e^ze^{-z}=1$.

Reverse the score to get the complementary probability:

For subtraction, put 1 over the denominator $1+e^{-z}$; the numerator becomes $(1+e^{-z})-1=e^{-z}$.

Shared Q2 uses $x$ instead of $z$; the identities are identical. Check: $S(0)=1/2$, $S(2)\approx0.8808$, and $S(-2)\approx0.1192$.

Logistic prediction

Add contributions, then apply the sigmoid. With $d$ features and fitted coefficients:

$x_j$ is feature $j$ of input $x$, $\widehat\beta_j$ its fitted weight, $\widehat\beta_0$ the fitted intercept, and $z$ the score. Hats mean estimated from data. $\widehat p_k(x)$ estimates the chance of class $k$; $e\approx2.718$.

Shared Q5: supplied coefficients are $-6,0.05,1$; supplied inputs are 40 study hours and GPA 3.5.

Estimated chance of an A: 37.8%. The probability is for this student; the training sample size is not an extra divisor.

Odds and logit

Odds compare the chance of class 1 with the chance of class 0. For probability $q=P(Y=1\mid X=x)$, with $0<q<1$:

$\ln$ is the natural logarithm. Logistic regression makes the log-odds a weighted score:

$d$ counts features, $x_j$ is feature $j$, $\beta_j$ its coefficient, and $\beta_0$ the intercept. Equivalently,

Shared Q6 proof: exponentiate $\ln(q/(1-q))=z$ to get $q/(1-q)=e^z$. Multiply by $1-q$ to get $q=e^z-e^zq$. Collect terms: $q(1+e^z)=e^z$. Divide by $1+e^z>0$ to obtain $q=e^z/(1+e^z)$. In reverse, $1-q=1/(1+e^z)$, so $q/(1-q)=e^z$ and its log is $z$.

Coefficient meaning: holding other features fixed, increasing $x_j$ by 1 adds $\beta_j$ to log-odds and multiplies odds by $e^{\beta_j}$. The probability change is not constant.

Numerical check: $q=0.8$ gives odds $0.8/0.2=4$, log-odds $\ln4\approx1.3863$, and the inverse gives $e^{\ln4}/(1+e^{\ln4})=4/5=0.8$.

Solving for a target probability

Turn the target chance back into a score, then solve a linear equation. Suppose one input $h$ can change while all others stay fixed:

$a$ is the intercept plus all fixed contributions, $b\neq0$ is the coefficient of $h$, and $q$ is the predicted class-1 probability. For target probability $0<t<1$:

$t$ is the requested probability, and $\ln$ is the natural logarithm. At $t=0.5$, $\ln(t/(1-t))=0$, so $h=-a/b$.

Shared Q5: keep GPA at 3.5. The fixed contribution is $a=-6+1(3.5)=-2.5$; the hours coefficient is $b=0.05$. For $t=0.5$:

Check: $z=-2.5+0.05(50)=0$, so $q=1/(1+e^0)=0.5$. If $b=0$, changing $h$ cannot change the model’s probability.

Probability thresholds

A probability becomes a label only after choosing a cutoff. Let $q=P(Y=1\mid X=x)$, with logistic score $z$ and sigmoid relation $q=1/(1+e^{-z})$. For threshold $0<t<1$:

$\widehat Y$ is the predicted label; $t$ is the minimum chance required to predict 1. Because the sigmoid increases with its score:

At $t=0.5$, predict 1 exactly when $z\geq0$. At equality, this convention assigns the tie to 1.

Invented example: $q=0.6$ predicts 1 at threshold 0.5 but predicts 0 at threshold 0.7. Changing the threshold changes the label, not the probability estimate.

Maximum likelihood

Fit coefficients that assign high probabilities to the observed labels. For independent training rows $i=1,\ldots,n$, let $y_i\in\{0,1\}$ be the actual label and $p_i=P(Y=1\mid X=x_i;\boldsymbol\beta)$ the proposed class-1 probability under coefficient vector $\boldsymbol\beta$.

For each row, use $p_i$ if $y_i=1$, or $1-p_i$ if $y_i=0$. Multiply the factors:

$n$ counts rows; $\prod$ means multiply. The exponent 0 makes the unused factor equal 1. Maximum likelihood estimation chooses

The hat marks fitted coefficients; “arg max” returns the coefficients giving the largest likelihood. Inputs and observed labels stay fixed while the candidate coefficients change.

Invented example: actual labels $1,0,1$ and proposed class-1 probabilities $0.8,0.3,0.6$ give

The middle row actually has label 0, so its factor is $1-0.3=0.7$.

Logistic coefficient tests

Compare the fitted effect with its uncertainty. To test coefficient $j$:

$\beta_j$ is the unknown population coefficient, $\widehat\beta_j$ its fitted estimate, and $\operatorname{SE}$ its estimated sample-to-sample variation. The test statistic $z_j$ is the estimate’s distance from zero in standard errors. This is distinct from the weighted score used to predict probabilities.

Under $H_0$ and suitable large-sample conditions, $z_j$ is approximately standard normal. The module uses the approximate two-sided rule:

Otherwise fail to reject, including at exactly $|z_j|=2$. Failure to reject does not prove zero; rejection does not prove causation.

Invented fit: $\widehat\beta_j=0.6$, $\operatorname{SE}=0.2$. Then $z_j=0.6/0.2=3$, so reject zero. In a multiple-input model, this tests the coefficient while the other modeled inputs are held fixed.

Multiclass logistic regression

One weighted score per class; divide each exponential by their total. For $K$ classes and $d$ input features:

$k$ selects a class, $j$ selects a feature, $\beta_{0k}$ is that class’s intercept, and $\beta_{jk}$ is its coefficient for feature $x_j$. The index $\ell$ includes every class in the denominator; $p_k(x)$ is the class-$k$ probability. $e$ is the exponential base. This conversion is called softmax.

Probabilities sum to 1. Predict the class with the largest $p_k(x)$, equivalently the largest score $z_k$. Fit coefficients by maximizing the product of probabilities assigned to actual training labels.

Invented scores: $0,\ln2,\ln3$ give weights $1,2,3$ and total 6. Probabilities are $1/6,2/6,3/6$; predict class 3.

Denominator check: with all $K$ classes in the sum, do not add another 1. In reference-class notation, one class has score 0 and weight $e^0=1$; its denominator can be written as 1 plus the exponentials for the other $K-1$ classes.