Two at once

Joint, marginal, and conditional distributions

Count people first

Start with a frequency example, then translate it into probabilities.

The game: flip a fair coin. Heads, you spin a dial that lands anywhere from 0 to 1. Tails, you spin a dial that lands anywhere from 0 to 2. Fair dial, no sticky spots. Imagine an illustrative group of 100 people split evenly between the two coin outcomes. In an actual random run the group counts fluctuate:

	how many	the dial lands	land below 1	average landing spot
heads	50	somewhere in 0–1	all 50	0.5 (middle of 0–1)
tails	50	somewhere in 0–2	about 25 (the bottom half of 0–2)	1.0 (middle of 0–2)
everyone	100		75	0.75

The group totals give two quantities we can compute before introducing the notation.

“What’s the chance the dial lands below 1?” → 75 out of 100 → 3/4.
“What’s the average landing spot?” → (50 × 0.5 + 50 × 1.0) ÷ 100 → 0.75.

Several useful terms describe parts of this example:

“Given heads” = look only at the heads row.
Marginal (or “unconditional”) = the everyone row — ignore the coin.
Joint = one cell: “heads AND below 1” is 50 people.
Weighted average = bigger groups count for more. If 90 people had gotten heads and 10 tails, the totals would lean toward the heads numbers.

Now use a discrete example to see the full joint-probability table.

One table over pairs

Two random quantities X and Y need one probability per pair of values. That table is the joint distribution: $p_{X,Y}(x, y) = \Pr(X = x,Y = y)$ — the chance that X lands on x and, in the same run, Y lands on y.

Flip a fair coin twice, independently. Let X = the first flip (1 for heads, 0 for tails) and Y = the total number of heads. Four equally likely outcomes — HH, HT, TH, TT — and each one drops into exactly one cell. Write the totals in the margins:

	Y = 0	Y = 1	Y = 2	row total
X = 0	1/4 (TT)	1/4 (TH)	0	2/4
X = 1	0	1/4 (HT)	1/4 (HH)	2/4
column total	1/4	2/4	1/4	1

All four quarters are in there and the corner says 1: the table slices the whole bag of outcomes by pair, nothing left out. The zeros are the interesting cells — a tails start (X = 0) makes two heads impossible. X and Y are tangled together: dependent.

Start with three moves on this table: add across a row, rescale a row, or compare cells to margins.

Collapse a row: the marginal

Only care about X? Add across each row and read the totals: 2/4 and 2/4 — the first flip is a fair coin, as it must be. Those row totals are the distribution of X alone, and they live in the table’s margin, so the jargon is the marginal distribution. That’s the entire meaning of the word: marginal = the totals written in the margin.

Same move down the columns: 1/4, 2/4, 1/4 is the marginal of Y — the head count on its own.

In symbols: $p_X(x)=\sum_y p_{X,Y}(x,y)$ , “add up the row — everything at x, whichever y came with it.” For a joint density, the analogous marginal is $f_X(x)=\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy$ : integrate along the y direction while holding x fixed.

Slice a row: the conditional

Told that X = 1 — the first flip came up heads? Delete every other row. What’s left is the X = 1 row, but it only sums to 2/4, and probabilities must sum to 1. So rescale: divide the row by its own total.

	Y = 0	Y = 1	Y = 2	total
the X = 1 row	0	1/4	1/4	2/4
÷ its total, 2/4	0	1/2	1/2	1

That bottom row is the conditional distribution $p_{Y\mid X}(y\mid 1)$ : given a heads start, the total is 1 or 2, fifty-fifty — the head you already have, plus one more fair flip. This is shrink the world in table form, and the formula just names the two numbers you used:

$p_{Y\mid X}(y\mid x) = p_{X,Y}(x, y) / p_X(x)$ — cell over row total, provided the row total is positive. Dividing by $p_X(x)$ is the rescale, because $p_X(x)$ is exactly the surviving row’s total.

Hold on to this move. In learning from data, the row you keep is “every output that can come back at one input,” and its center is the entire object of study.

Go deeper: Bayes — slicing the other way

With the full table in hand you never need Bayes’ rule: to get $p_{X\mid Y}(x\mid y)$ , slice a column and divide by its total. Bayes’ rule is for when you don’t have the table — you were handed row slices and margins, and you want a column slice.

Where it comes from: the same cell, written in two orders — $p_{X,Y}(x, y) = p_{Y\mid X}(y\mid x) \times p_X(x) = p_{X\mid Y}(x\mid y) \times p_Y(y)$ . Set the two equal and divide by $p_Y(y)$ :

p_{X\mid Y}(x\mid y) = p_{Y\mid X}(y\mid x) \times p_X(x) / p_Y(y)

Rebuild the cell from the slices you have, then divide by the total of the slice you want — that rescaling is all Bayes’ rule is.

Conditioning on an exact continuous input

There is a subtlety before using this idea in ML. For a continuously distributed X, $\Pr(X=x)=0$ . You cannot plug that zero into the ordinary event-conditioning formula and divide by it.

For jointly distributed variables with a suitable joint density, instead take a vertical slice of the density and normalize it:

f_{Y\mid X}(y\mid x)=\frac{f_{X,Y}(x,y)}{f_X(x)},\qquad f_X(x)>0.

Why does this work? The integral of the numerator over y is exactly $f_X(x)$ . Dividing by it makes the conditional slice integrate to one. These are densities, not probabilities of exact points.

For smooth examples, picture conditioning first on a narrow strip around x. As the strip narrows, the normalized distribution of Y approaches the conditional density. This provides a useful interpretation of conditioning at a point without dividing by a zero event probability.

For example, a joint density of 1/2 on the rectangle $0\le x\le1$ , $0\le y\le2$ has marginal density $f_X(x)=1$ . The conditional density of Y at any interior x is $(1/2)/1=1/2$ on [0,2]. Its conditional mean is 1.

For a joint model at an input where $f_X(x)=0$ , this ratio does not define a conditional density. No measure-theory machinery is needed for the positive-density examples in this module.

What independence looks like

Now the untangled case. Flip a fair coin (X is 0 or 1) and roll a fair 4-sided die (Y is 1 to 4), with neither touching the other:

	Y = 1	Y = 2	Y = 3	Y = 4	row total
X = 0	1/8	1/8	1/8	1/8	1/2
X = 1	1/8	1/8	1/8	1/8	1/2
column total	1/4	1/4	1/4	1/4	1

Check any cell against its margins: $1/8 = 1/2 \times 1/4$ , every time. That is independence as a table: $p_{X,Y} = p_X \times p_Y$ in every cell — the table holds no information beyond its margins. Slice any row and rescale it: you get the column totals back, so news about X moves nothing about Y.

The coin-pair table fails this test: the cell (X = 1, Y = 0) holds 0, but row total × column total = $2/4 \times 1/4 = 1/8$ . One failed cell is all it takes — dependent.

For the continuous version, a joint density $f_{X,Y}(x, y)$ is a surface over the plane, and the volume under the surface over a region = the chance of landing in that region.

Try the moves

Do they miss their centers together?

Write $\mathbb E[X]$ for the average of X — its long-run mean, from the average. Covariance asks whether X and Y miss their averages together:

\operatorname{Cov}[X, Y] = \mathbb E[(X - \mathbb E[X])(Y - \mathbb E[Y])] = \mathbb E[XY] - \mathbb E[X] \times \mathbb E[Y]

Read the shortcut as: the average of the product, minus the product of the averages. Positive: when X comes in high, Y tends to come in high too. Negative: they miss on opposite sides. Zero: “uncorrelated.”

For the coin pair: $\mathbb E[X] = 1/2$ , $\mathbb E[Y] = 1$ , and $\mathbb E[XY]$ takes one pass over the table — multiply each cell by its x × y and add; only the cells with x = 1 survive: $1\times 1\times (1/4) + 1\times 2\times (1/4) = 3/4$ . So $\operatorname{Cov}[X, Y] = 3/4 - 1/2 \times 1 = 1/4$ — positive, as it should be: a heads start pushes the total up.

One warning: uncorrelated is weaker than independent. Independence kills all patterns between X and Y; zero covariance only kills the straight-line one. Two lines make the gap vivid. Let X be −1, 0, or 1, a third each, and Y = X². Then $\mathbb E[XY] = \mathbb E[X^3] = (-1 + 0 + 1)/3 = 0$ and $\mathbb E[X] = 0$ , so $\operatorname{Cov}[X, Y] = 0 - 0 \times \mathbb E[Y] = 0$ — perfectly uncorrelated. Yet Y is completely determined by X: $\Pr(Y = 0) = 1/3$ , but told that X = 1, $\Pr(Y = 0\mid X = 1) = 0$ . No straight-line lean, total dependence.

Condition, then average

Extend the earlier two-stage experiment by replacing the coin with a die. There are now six cases to condition on, each with its own probability.

Roll a fair 6-sided die and call the result D. Then draw U uniformly from the interval [0, D] — a random point on a bar whose length the die just chose. Two questions, one move: freeze the die, answer the easy question, then average the six answers, each weighted by 1/6.

(a) What is $\Pr(U \le 1)$ — the chance U is at most 1? Freeze D = d: U is uniform on a bar of length d, and the slice [0, 1] is $1/d$ of that bar, so $\Pr(U \le 1\mid D = d) = 1/d$ . Now average the six cases:

\begin{aligned} \Pr(U\le1)&=\frac16\left(1+\frac12+\frac13+\frac14+\frac15+\frac16\right)\\ &=\frac16\cdot\frac{60+30+20+15+12+10}{60}\\ &=\frac16\cdot\frac{147}{60}=\frac{49}{120}\approx0.4083. \end{aligned}

(b) What is E[U]? Freeze D = d: a uniform draw from [0, d] averages to the midpoint, $\mathbb E[U\mid D = d] = d/2$ . Average those answers:

\mathbb E[U]=\frac16\left(\frac12+\frac22+\frac32+\frac42+\frac52+\frac62\right)=\frac16\cdot\frac{21}2=\frac74.

Read those formulas as expected counts in a balanced illustration: imagine 600 people, 100 per face. Face 1 sends all 100 below 1; face 2 sends 50; then about 33, 25, 20, 17 — add down the column: about 245 of 600, the same 0.41. $\Pr(D = d)$ appears in a formula only to say how big each group is; $\Pr(U \le 1\mid D = d)$ only to say what happens inside that group. Group size × what happens inside = one cell of people; adding the cells = the answer.

These are the law of total probability and the law of total expectation:

$\Pr(U\le1)=\sum_d\Pr(U\le1\mid D=d)\Pr(D=d).$

$\mathbb E[U]=\sum_d\mathbb E[U\mid D=d]\Pr(D=d).$

The conditional expectation $\mathbb E[U\mid D = d]$ is a function of d — here, d/2. Conditioning turns one hard two-variable question into six easy one-variable ones; averaging glues the answers back together.

Stacking up

Stack n of these into one vector and the same machinery scales: a mean per entry, a covariance per pair — an n-by-n grid of them, the object you will meet again the moment you touch data in bulk.

Everything on one page: the cheat sheet.