7. The law of total probability

Sometimes the cleanest way to crack a hard probability is to break it into simpler cases first. That is the whole idea behind the law of total probability.

Partitions

Start by splitting the entire sample space into a handful of cases — call them B_1, B_2, B_3. To be a partition they must:

• cover everything: B_1 \cup B_2 \cup B_3 = \Omega, and
• not overlap: B_i \cap B_j = \varnothing for i \ne j.

Exactly one of them happens, always.

Now suppose some event A cuts across several of these cases at once, showing up partly in each. A lives partly in B_1, partly in B_2, partly in B_3.

The formula

Since the B_i don't overlap, the pieces of A inside them don't either. So A is the disjoint union of those pieces, and Axiom 3 lets us just add:

P(A) = P(A \cap B_1) + P(A \cap B_2) + P(A \cap B_3)

Now rewrite each piece with the multiplication rule, $P(A \cap B_i) = P(A \mid B_i)P(B_i)$:

\boxed{\;P(A) = \sum_{i} P(A \mid B_i)\,P(B_i)\;}

Read it as: A's share of a case, weighted by how likely that case is, summed over every case. It's a weighted average of conditional probabilities, with the weights being the case probabilities (which sum to 1).

That last observation is a free sanity check: since P(A) is a weighted average of the P(A \mid B_i), it must land between the smallest and largest of them. If your answer falls outside that range, you've made an arithmetic error.

Worked example: the factory

A factory has three machines.

What fraction of all bulbs are defective?

The machines partition the output — every bulb comes from exactly one. Weight each defect rate by its share:

P(D) = (0.5)(0.02) + (0.3)(0.03) + (0.2)(0.05) = 0.010 + 0.009 + 0.010 = 0.029

2.9% overall. Notice the sanity check: the individual rates run from 2% to 5%, and 2.9% sits inside that range, closer to the low end because the low-rate machine makes half the bulbs.

Machines 1 and 3 contribute equally to the defect pile (0.010 each) despite machine 3 being 2.5× as defect-prone — because machine 1 makes 2.5× as many bulbs. That trade-off is the whole content of the law.

It was hiding in Bayes all along

Remember Bayes' theorem, and that denominator P(B) — the overall chance of the evidence? Expand it over the partition \{A, A^c\}:

P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B \mid A)P(A) + P(B \mid A^c)P(A^c)}

The denominator is exactly a law-of-total-probability sum over two cases. In the medical-test lesson we computed

P(+) = (0.99)(0.01) + (0.01)(0.99) = 0.0198

and that was a total-probability sum all along. This is the standard way the two laws are used together: total probability builds the denominator, Bayes does the flip.

Doing it in Python

The law is a dot product — cases weighted by their probabilities:

def total_probability(cases):
    """cases: {name: (P(case), P(A | case))} -> P(A)."""
    return sum(weight * cond for weight, cond in cases.values())

machines = {
    "M1": (0.5, 0.02),
    "M2": (0.3, 0.03),
    "M3": (0.2, 0.05),
}

# The weights must be a genuine partition.
assert abs(sum(w for w, _ in machines.values()) - 1) < 1e-12

p_defect = total_probability(machines)
print("P(defective) =", round(p_defect, 4))

# And which machine made a given defective bulb? That's Bayes, using the
# number we just built as the denominator.
for name, (weight, cond) in machines.items():
    print(f"P({name} | defective) = {weight * cond / p_defect:.4f}")

That second loop is worth staring at, because the answer is not the one most people predict. Machine 3 has the worst defect rate but makes the fewest bulbs, and those two effects cancel almost exactly: 0.5 \times 0.02 and 0.2 \times 0.05 are the same number. So M3 ends up tied with M1 as the most likely source of a given defective bulb — not the least. The least likely is M2, which is in the middle on both volume and defect rate. What ranks the machines is the product, never the defect rate alone.

Your turn

1. A bag has two coins: one fair, one two-headed. You pick one at random and flip it. What's the probability of heads?

2. In the same setup, you flip and get heads. What's the probability you picked the two-headed coin?

3. 60% of students take statistics; 80% of them pass a general numeracy test. Of the other 40%, only 50% pass. What fraction of all students pass?

Solutions

1. Partition on which coin you picked, each with probability 1/2:

P(H) = P(H \mid \text{fair})P(\text{fair}) + P(H \mid \text{2-headed})P(\text{2-headed}) = (0.5)(0.5) + (1)(0.5) = 0.25 + 0.5 = 0.75

Three quarters. Between 0.5 and 1 as the sanity check requires.

2. Now flip the conditioning with Bayes, using the 0.75 we just built:

P(\text{2-headed} \mid H) = \frac{P(H \mid \text{2-headed})P(\text{2-headed})}{P(H)} = \frac{(1)(0.5)}{0.75} = \frac{2}{3}

One head raises the two-headed coin from 1/2 to 2/3. Not conclusive — the fair coin lands heads often enough that a single flip is weak evidence. (Flip it five more times and get heads every time, and the posterior climbs above 98%.)

3. Partition on whether the student takes statistics:

P(\text{pass}) = (0.8)(0.6) + (0.5)(0.4) = 0.48 + 0.20 = 0.68

68%. Between 50% and 80%, leaning toward 80% because the higher-passing group is the larger one.

Check yourself in code

Compute the overall defect rate for the three-machine factory, then use it as the Bayes denominator to find where a defective bulb probably came from.

Print exactly this:

P(defective) = 0.029
M1 0.3448
M2 0.3103
M3 0.3448

Round the overall probability to 4 decimal places and print each posterior to 4 decimal places, one machine per line.

machines = {
    "M1": (0.5, 0.02),
    "M2": (0.3, 0.03),
    "M3": (0.2, 0.05),
}

p_defect = sum(w * c for w, c in machines.values())
print("P(defective) =", round(p_defect, 4))

# For each machine print its name and P(machine | defective), to 4 decimals.

machines = {
    "M1": (0.5, 0.02),
    "M2": (0.3, 0.03),
    "M3": (0.2, 0.05),
}

p_defect = sum(w * c for w, c in machines.values())
print("P(defective) =", round(p_defect, 4))

for name, (weight, cond) in machines.items():
    print(name, f"{weight * cond / p_defect:.4f}")

Look at what came back: M1 and M3 are tied at 0.3448. A defective bulb is exactly as likely to have come from the machine that makes half the output at a 2% defect rate as from the one making a fifth at 5% — because 0.5 \times 0.02 and 0.2 \times 0.05 are the same number. The weighted contributions are what matter, not the defect rates on their own.

(If you tried max() to pick a single "most likely" machine, you'd get an answer decided by floating-point rounding on a genuine tie. Worth remembering whenever you rank probabilities in code.)

Split the sample space into non-overlapping cases. Weight each case by how likely it is. Then add up A's share of every one. That's the whole law — and it is the engine inside every Bayes denominator you'll ever write.

That closes the foundations. Next section: random variables, where outcomes stop being labels like "heads" and become numbers we can average.