Conditional probability and Bayes' theorem · Probability & Statistics

What is the probability of A, given that B already happened?

That question — updating what you believe once you learn something new — is one of the most useful ideas in all of probability, and the one most often got wrong in practice.

Conditioning shrinks the sample space

Picture the whole sample space as a box. Inside it, event B is one region, and event A is another region that partly overlaps it.

Here is the key move. Saying "given B happened" means we throw away everything outside B. B becomes our entire world — the sample space has shrunk to just the inside of B.

So the only question left is: within that shrunken world, what fraction of B also lies in A? That fraction — the overlap divided by all of B — is the definition:

$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0$

The denominator is doing the renormalising. We've restricted attention to B, so probabilities have to be rescaled to sum to 1 again over the smaller space.

P(B) > 0 is not a technicality to skip. You cannot condition on something that cannot happen — the fraction is undefined. Conditioning on probability-zero events can be made sense of, but it needs machinery we don't have until §9.

The multiplication rule

Rearranging is trivial but the result gets used constantly:

$P(A \cap B) = P(A \mid B)\,P(B)$

"The chance both happen is the chance B happens, times the chance A happens given it did."

Bayes' theorem

Now watch a small piece of algebra that turns out to be enormously useful.

There is nothing special about the order of A and B. Condition on A instead and you get the same joint probability the other way round:

$P(A \cap B) = P(B \mid A)\,P(A)$

Both expressions equal the very same quantity, so set them equal:

$P(A \mid B)\,P(B) = P(B \mid A)\,P(A)$

Divide through by P(B):

$\boxed{\;P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}\;}$

That's Bayes' theorem. All it does is let you flip which event you're conditioning on — turning a P(\text{evidence} \mid \text{hypothesis}), which is often easy to measure, into a P(\text{hypothesis} \mid \text{evidence}), which is what you actually want to know.

The pieces have names you'll meet again in §7:

Term	Name	Means
P(A)	prior	belief before the evidence
P(B \mid A)	likelihood	how well A predicts the evidence
P(A \mid B)	posterior	belief after the evidence
P(B)	evidence	how likely the evidence was overall

The result that surprises everyone

Suppose a disease affects 1% of people. There's a test that correctly flags 99% of the people who have it, and wrongly flags 1% of those who don't.

You take the test. It comes back positive. How likely are you to have the disease?

Most people's gut says 99%. Let's compute it.

Write D for "has the disease" and + for "tests positive". We're given P(D) = 0.01, P(+ \mid D) = 0.99, P(+ \mid D^c) = 0.01.

First the denominator — the overall chance of testing positive, which collects both routes to a positive result:

$P(+) = P(+ \mid D)P(D) + P(+ \mid D^c)P(D^c) = (0.99)(0.01) + (0.01)(0.99) = 0.0198$

Now Bayes:

$P(D \mid +) = \frac{P(+ \mid D)\,P(D)}{P(+)} = \frac{0.0099}{0.0198} = \frac{1}{2}$

Exactly one half. A test that sounds 99% accurate, and a positive result is still nothing more than a coin flip.

The same thing, counting real people

If that feels impossible, drop the decimals and count 10,000 people.

1% are sick: 100 people. The other 9,900 are healthy.
Of the 100 sick, the test correctly catches 99.
Of the 9,900 healthy, 1% are wrongly flagged: also 99.

	Tests +	Tests −	Total
Sick	99	1	100
Healthy	99	9,801	9,900
Total	198	9,802	10,000

There it is. 99 positives come from the sick and 99 from the healthy — the same number from each group. Among everyone who tests positive, half are genuinely sick:

$\frac{99}{198} = \frac{1}{2}$

Exactly what Bayes gave us, and now you can see why. The disease is rare, so even a small false-positive rate applied to the enormous healthy pool produces as many positives as the accurate test applied to the tiny sick pool.

The lesson: never read a test result without first asking how rare the thing being tested for is. That rarity is the prior, and ignoring it is called the base rate fallacy.

Doing it in Python

Bayes is short enough to write literally:

def bayes(prior, likelihood, false_positive):
    """P(D | +) from P(D), P(+|D) and P(+|not D)."""
    evidence = likelihood * prior + false_positive * (1 - prior)
    return likelihood * prior / evidence

print("P(disease | positive) =", bayes(0.01, 0.99, 0.01))

# The base rate is what drives it. Same test, different prevalence:
for prevalence in (0.001, 0.01, 0.1, 0.5):
    p = bayes(prevalence, 0.99, 0.01)
    print(f"prevalence {prevalence:>5}: P(sick | +) = {p:.4f}")

Run that second loop and the whole lesson is visible in four lines: the test never changed, only the population did, and the answer moved from 9% to 99%.

Your turn

1. A box has 3 red and 2 blue balls. You draw two without replacement. What's P(\text{both red})?

2. Using the same 1%-disease test, what's the probability you're healthy given a negative result?

3. A factory's screws come from two machines: A makes 70% with a 2% defect rate, B makes 30% with a 5% defect rate. A screw is defective — which machine more likely made it?

Solutions

1. Use the multiplication rule. The first draw is red with probability 3/5. Given that, four balls remain and two are red, so the second is red with probability 2/4:

$P(\text{both red}) = \frac{3}{5} \times \frac{2}{4} = \frac{6}{20} = 0.3$

The conditioning is the whole point of "without replacement" — the second probability genuinely depends on the first outcome.

2. From the table: 9,801 of the 9,802 people who test negative are healthy.

$P(D^c \mid -) = \frac{9801}{9802} \approx 0.9999$

So a negative result is extremely informative and a positive one barely is. That asymmetry is typical of screening for rare conditions, and it's why a positive screen is normally followed by a second, different test rather than treatment.

3. Let M_A, M_B be the machines and D "defective".

$P(D) = (0.02)(0.7) + (0.05)(0.3) = 0.014 + 0.015 = 0.029$

$P(M_A \mid D) = \frac{0.014}{0.029} \approx 0.483 \qquad P(M_B \mid D) = \frac{0.015}{0.029} \approx 0.517$

Machine B, but only just — 51.7% against 48.3%. B is more than twice as defect-prone, yet A makes so many more screws that the two nearly cancel. Once again the base rate does most of the work.

Check yourself in code

Build the disease table from scratch by counting people, and confirm it agrees with Bayes' theorem.

Print exactly this:

sick and positive: 99
healthy and positive: 99
P(sick | positive) = 0.5
matches Bayes: True

Use a population of 10,000, prevalence 1%, sensitivity 99%, false positive rate 1%. Round the probability to 4 decimal places.

population = 10_000
prevalence, sensitivity, false_positive = 0.01, 0.99, 0.01

sick = population * prevalence
healthy = population - sick

sick_pos = sick * sensitivity
print("sick and positive:", round(sick_pos))

# Count the healthy who test positive, then take the fraction of all positives
# that are genuinely sick, and compare it with Bayes' theorem.

population = 10_000
prevalence, sensitivity, false_positive = 0.01, 0.99, 0.01

sick = population * prevalence
healthy = population - sick

sick_pos = sick * sensitivity
healthy_pos = healthy * false_positive

print("sick and positive:", round(sick_pos))
print("healthy and positive:", round(healthy_pos))

by_counting = sick_pos / (sick_pos + healthy_pos)
print("P(sick | positive) =", round(by_counting, 4))

evidence = sensitivity * prevalence + false_positive * (1 - prevalence)
by_bayes = sensitivity * prevalence / evidence
print("matches Bayes:", round(by_counting, 4) == round(by_bayes, 4))

Conditioning shrinks the sample space. Bayes' theorem flips the conditioning. And the base rate decides how much the evidence is really worth.

Next: the opposite situation — when learning B tells you nothing about A.

5. Conditional probability and Bayes' theorem