Bayesian vs. frequentist · Probability & Statistics

We've now built two complete frameworks for inference. This lesson puts them side by side — what actually differs, what doesn't, and when the choice changes your answer.

The core disagreement

It comes down to a single question: what is allowed to have a probability?

Frequentist: probability is long-run frequency. Only repeatable events have probabilities. A parameter is a fixed constant — it doesn't repeat, so it has no distribution. You can only make probability statements about the data, and hence about procedures.

Bayesian: probability is degree of belief. Anything uncertain can have a probability, including a parameter, a hypothesis, or a one-off event. You make probability statements about \theta directly.

This is exactly the interpretive split from §0's lesson on what probability means — the frequentist and subjective readings — resurfacing as a methodological difference. It was never going to stay abstract.

Side by side

	Frequentist	Bayesian
\theta	fixed constant	random variable
Data	random	fixed (once observed)
Prior	none	required
Output	point estimate + CI, p-value	full posterior distribution
Interval	95% of intervals cover \theta	95% probability \theta is here
Answers	P(\text{data} \mid \theta)	P(\theta \mid \text{data})
Nuisance parameters	profile or plug in	integrate out
Optimality	minimax, unbiasedness, efficiency	expected loss under the posterior

What they agree on

More than the argument suggests.

With lots of data, they agree. The Bernstein–von Mises theorem says the posterior converges to a Normal centred at the MLE with variance 1/I_n(\theta) — exactly the frequentist sampling distribution from §4. Credible and confidence intervals become numerically almost identical.

With a flat prior, MAP = MLE. Maximising the posterior reduces to maximising the likelihood.

The likelihood is shared. Both frameworks use the same $p(\text{data} \mid \theta)$. They differ in what they do with it, not in how they model the data.

Regularization is a prior. Ridge is MAP under a Normal prior; lasso is MAP under a Laplace prior (§6). Many frequentist methods are Bayesian procedures wearing different clothes.

Neither is assumption-free. A frequentist chooses a model, a test statistic, a stopping rule, an \alpha. A Bayesian chooses those and a prior. The difference is which assumptions are written down.

Where they genuinely diverge

1. Small samples. With little data the prior dominates, so the frameworks give different answers. The Bayesian answer is better if the prior is good and worse if it isn't. This is where the choice matters most.

2. The likelihood principle. Bayesian inference depends only on the observed likelihood. Frequentist inference depends on what might have happened — including the experimenter's intentions.

The classic illustration: you observe 9 heads in 12 flips.

If you had fixed n = 12, the data is Binomial.
If you had flipped until 3 tails appeared, the data is Negative Binomial.

The two p-values differ, because "at least as extreme" means different things under the two designs. Same data, same likelihood function, different conclusions, differing only in the experimenter's stopping rule. The Bayesian posterior is identical in both cases, because the two likelihoods differ only by a constant that cancels in normalisation.

Whether this is a fatal flaw of frequentism or a sensible acknowledgement that design matters is one of the genuinely open arguments in statistics.

3. Optional stopping. Related, and practically important. Peeking at frequentist p-values as data arrives and stopping when significant inflates the Type I error rate dramatically — with enough peeks you can reach p < 0.05 almost surely under a true null. A Bayesian posterior can be inspected at any time without any such correction.

(This is not a free lunch. Stopping when the posterior looks good still biases reported effect sizes, and the guarantee is about coherence, not about frequentist error control.)

4. Nuisance parameters. Bayesians integrate them out, which is clean and automatic. Frequentists profile or plug in, which is harder and sometimes produces poor answers.

5. Prior information. If you genuinely know something — previous trials, physical limits — the Bayesian framework uses it. The frequentist framework has no formal slot for it.

Which to use

The honest answer is that this is largely settled in practice, and the answer is both.

Frequentist tends to suit: regulated settings needing guaranteed error rates (clinical trials), large datasets where priors don't matter, and situations where a defensible "objective" analysis is required.

Bayesian tends to suit: small samples with real prior knowledge, hierarchical models, sequential decisions, and any case where you need P(\text{hypothesis} \mid \text{data}) to make a decision.

In practice most working statisticians use both, choosing by problem rather than by ideology. Modern computation removed the practical barrier that once made Bayesian methods infeasible, and the philosophical argument is much quieter than it was.

Worked example

A coin: 9 heads in 12 flips. Is it biased toward heads?

Frequentist. H_0: p = 0.5 against H_1: p > 0.5.

$p\text{-value} = P(X \ge 9 \mid p = 0.5) = \sum_{k=9}^{12}\binom{12}{k}(0.5)^{12} = 0.073$

At \alpha = 0.05: fail to reject. Not significant. And note what you may not conclude — nothing about P(p > 0.5).

Bayesian, with a uniform Beta(1,1) prior:

$p \mid \text{data} \sim \text{Beta}(10, 4)$

$E[p \mid \text{data}] = \frac{10}{14} = 0.714, \qquad P(p > 0.5 \mid \text{data}) = 0.954$

A 95.4% probability the coin favours heads.

These sound contradictory but aren't — they answer different questions. The p-value asks "how surprising is this data if the coin is fair?" (answer: mildly). The posterior asks "how likely is the coin to favour heads?" (answer: quite).

The numbers are close (0.073 and 1 - 0.954 = 0.046) because with a flat prior the two calculations are nearly mirror images. They are not the same quantity, and the near-agreement here is a consequence of the flat prior, not a general rule.

Change the prior to Beta(50,50) — a strong belief the coin is fair, as it would be for a coin from a bank — and P(p > 0.5 \mid \text{data}) drops substantially. Same data, different conclusion, because the prior information was different. That is the framework working as designed, not a defect.

Doing it in Python

Both analyses of the same data:

import numpy as np
from scipy.stats import binom, beta

k, n = 9, 12

# Frequentist: one-sided exact binomial test
p_value = 1 - binom.cdf(k - 1, n, 0.5)
print(f"frequentist p-value       : {p_value:.4f}")
print(f"  reject at alpha = 0.05? : {p_value < 0.05}")
print(f"  MLE                     : {k/n:.4f}")

# Bayesian with a uniform prior
post = beta(1 + k, 1 + n - k)
print(f"\nBayesian posterior        : Beta({1+k}, {1+n-k})")
print(f"  posterior mean          : {post.mean():.4f}")
print(f"  P(p > 0.5 | data)       : {1 - post.cdf(0.5):.4f}")
print(f"  95% credible interval   : ({post.ppf(0.025):.4f}, {post.ppf(0.975):.4f})")

How the prior changes the Bayesian conclusion — and how the data eventually wins:

from scipy.stats import beta

priors = {
    "uniform Beta(1,1)":        (1, 1),
    "mild fair Beta(5,5)":      (5, 5),
    "strong fair Beta(50,50)":  (50, 50),
    "sceptic Beta(200,200)":    (200, 200),
}

for label, (k, n) in [("9 of 12", (9, 12)), ("90 of 120", (90, 120)),
                      ("900 of 1200", (900, 1200))]:
    print(f"\ndata: {label}")
    for name, (a0, b0) in priors.items():
        post = beta(a0 + k, b0 + n - k)
        print(f"  {name:>24}: P(p > 0.5) = {1 - post.cdf(0.5):.4f}")

print("\nAt n=12 the prior dominates. By n=1200 every prior agrees.")

The likelihood principle violation, made concrete:

from scipy.stats import binom, nbinom, beta

k, n = 9, 12          # 9 heads, 3 tails

# Design A: flip exactly 12 times, count heads
p_binomial = 1 - binom.cdf(k - 1, n, 0.5)

# Design B: flip until 3 tails appear, count heads along the way
p_negbin = 1 - nbinom.cdf(k - 1, 3, 0.5)

print("same data: 9 heads and 3 tails\n")
print(f"  p-value if n was fixed at 12       : {p_binomial:.4f}")
print(f"  p-value if we flipped until 3 tails: {p_negbin:.4f}")
print(f"  the two differ by a factor of {max(p_binomial, p_negbin)/min(p_binomial, p_negbin):.2f}")

post = beta(1 + k, 1 + n - k)
print(f"\n  Bayesian P(p > 0.5) is {1 - post.cdf(0.5):.4f} under BOTH designs,")
print("  because the two likelihoods differ only by a constant.")

Optional stopping — where the frameworks behave very differently:

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(0)
trials, max_n, alpha = 3_000, 200, 0.05

# H0 is TRUE in every run. How often does peeking find "significance"?
fixed_hits = peek_hits = 0
for _ in range(trials):
    x = rng.standard_normal(max_n)
    ns = np.arange(1, max_n + 1)
    z = np.cumsum(x) / np.sqrt(ns)
    p = 2 * (1 - norm.cdf(np.abs(z)))
    fixed_hits += p[-1] < alpha                 # look once, at the end
    peek_hits += (p[10:] < alpha).any()         # look after every observation

print(f"look once at n={max_n}      : false positive rate {fixed_hits/trials:.4f}")
print(f"peek after every observation: false positive rate {peek_hits/trials:.4f}")
print("\nPeeking destroys the frequentist guarantee. A Bayesian posterior")
print("can be inspected at any time without a correction.")

Your turn

1. A frequentist CI is (2,8) and a Bayesian credible interval is $(2.1, 7.9)$. Do they mean the same thing?

2. When would a Bayesian and frequentist analysis give very different answers?

3. Is the prior a weakness of Bayesian statistics?

Solutions

1. No — the numbers are similar but the statements are different in kind.

The confidence interval says: the procedure that generated (2,8) captures the true \theta in 95% of repeated samples. About this interval, nothing probabilistic can be said — \theta is either in it or not.

The credible interval says: given the prior and this data, $P(2.1 < \theta < 7.9) = 0.95$. That is a claim about \theta in this specific analysis.

They're numerically close here because the data is presumably informative enough that the prior barely matters — the Bernstein–von Mises regime. Similar numbers, different meanings, and the difference matters when you report conclusions.

2. Chiefly when the data is weak relative to the prior:

Small samples. With n = 5 the prior can dominate entirely.
Rare events. Zero successes in 20 trials gives an MLE of exactly 0 — a claim of impossibility — while any sensible prior gives a small positive estimate.
Genuinely informative priors. Physical constraints, or a large body of previous trials, will pull the Bayesian answer far from the MLE.
Boundary problems. When the MLE sits on the edge of the parameter space (a variance component of zero, perfect separation in logistic regression), frequentist asymptotics break down while the posterior stays well behaved.
Complex hierarchical models, where frequentist methods struggle with nuisance parameters that Bayesians simply integrate out.

Conversely, with large n and a reasonable prior they agree closely, and arguing about which to use is not the best use of anyone's time.

3. It's a trade-off, not a straightforward weakness — and the framing of the question is worth resisting.

The case against: it's subjective. Two analysts get different answers from the same data. In a regulatory or adversarial setting that's a real problem, and "choose a prior that gives the answer I want" is an available abuse.

The case for: every analysis makes assumptions. Choosing a model, a test statistic, a significance level, a stopping rule, which outliers to drop — these are all judgement calls, and frequentist practice simply leaves them unlabelled. The Bayesian prior is an assumption written down explicitly, where it can be criticised and varied.

In practice the argument is defused by two things: with enough data the prior stops mattering, and sensitivity analysis — reporting results under several priors, as the code above does — shows exactly how much the conclusion depends on it. If the answer is stable across reasonable priors, the objection has no force. If it isn't, that's important information you'd otherwise have missed.

Check yourself in code

Analyse the same coin data both ways and confirm they answer different questions.

Print exactly this:

frequentist p 0.073
significant: False
posterior mean 0.7143
P(p > 0.5) 0.9539

Use 9 heads in 12 flips, a one-sided exact binomial test against p = 0.5, and a uniform Beta(1,1) prior. Round the p-value to 4 decimal places, the posterior mean to 4, and the probability to 4.

from scipy.stats import binom, beta

k, n = 9, 12

p_value = 1 - binom.cdf(k - 1, n, 0.5)
print("frequentist p", round(p_value, 4))
print("significant:", bool(p_value < 0.05))

# Build the Beta(1+k, 1+n-k) posterior and report its mean and P(p > 0.5).

from scipy.stats import binom, beta

k, n = 9, 12

p_value = 1 - binom.cdf(k - 1, n, 0.5)
print("frequentist p", round(p_value, 4))
print("significant:", bool(p_value < 0.05))

post = beta(1 + k, 1 + n - k)
print("posterior mean", round(float(post.mean()), 4))
print("P(p > 0.5)", round(float(1 - post.cdf(0.5)), 4))

The two frameworks disagree about what may have a probability: a frequentist parameter is fixed, a Bayesian one is random. That produces different objects — P(\text{data} \mid \theta) versus P(\theta \mid \text{data}) — and different guarantees. They converge with enough data, they share the likelihood, and many frequentist methods turn out to be Bayesian procedures in disguise. The practical differences bite in small samples, at boundaries, and under optional stopping.

Next: the Bayesian answer to hypothesis testing itself.

48. Bayesian vs. frequentist