Nonparametric tests · Probability & Statistics

Every test so far assumed a distribution — usually Normality. Nonparametric tests drop that assumption, working instead with signs and ranks.

The trade is straightforward: fewer assumptions, slightly less power when the assumptions would have held, and much better behaviour when they wouldn't.

When to reach for them

Small samples where you can't check Normality and can't rely on the CLT.
Obviously skewed data or clear outliers.
Ordinal data — rankings, Likert scales — where the numbers have order but no meaningful arithmetic. The "average" of agree and strongly disagree is not a quantity.
When you want a conclusion that doesn't depend on a distributional choice.

The sign test

The simplest of all. For paired data, count how many differences are positive.

Under H_0 (the median difference is 0), each pair is equally likely to go either way, so

$S = \#\{d_i > 0\} \sim \text{Binomial}(n, 0.5)$

Ties (zero differences) are dropped and n reduced accordingly.

That's the whole test — it uses only the sign of each difference, not its size. That makes it extraordinarily robust: an outlier of +1000 counts exactly the same as +0.1.

It's also the source of its weakness. Throwing away magnitudes throws away information, so it has the lowest power of the tests here (ARE of $2/\pi \approx 0.64$ against the t-test for Normal data).

Note it tests the median, not the mean.

Wilcoxon signed-rank test

A middle ground: use the ranks of the absolute differences, not just their signs.

Compute the differences d_i; drop zeros.
Rank |d_i| from smallest to largest.
Attach the original signs to those ranks.
Sum the positive ranks: W^+.

Large or small W^+ is evidence against H_0.

This uses more information than the sign test — bigger differences carry more weight — while remaining robust, because an extreme value only ever gets the top rank, not an unbounded score.

It assumes the differences are symmetrically distributed around the median, which the sign test does not need.

ARE against the t-test is 3/\pi \approx 0.955 for Normal data — you give up about 5% efficiency. For heavy-tailed data it can be far more powerful than the t-test.

Mann–Whitney U (Wilcoxon rank-sum)

The nonparametric counterpart of the two-sample t-test.

Pool both groups and rank all N = n_1 + n_2 observations together.
Sum the ranks belonging to group 1: R_1.
Compute

$U_1 = R_1 - \frac{n_1(n_1+1)}{2}$

U has a neat interpretation: it counts how many pairs (x, y) have x > y across the two groups. So the test is really asking

$H_0: P(X > Y) = 0.5$

Which is not quite "the medians are equal". If the two distributions have the same shape and differ only by a shift, it is a test of medians. If the shapes differ — one much more spread out than the other — a significant result may reflect the difference in shape rather than location. Worth knowing before you report it as a median comparison.

ARE against the t-test is again 3/\pi \approx 0.955 for Normal data.

Kruskal–Wallis

The nonparametric one-way ANOVA: rank all observations across k groups and compare the average ranks. The statistic is approximately \chi^2_{k-1}.

Same relationship to Mann–Whitney as ANOVA has to the t-test — and same limitation: a significant result says the groups differ somewhere, without saying where.

The map

Parametric	Nonparametric	Compares
One-sample t	Sign, Wilcoxon signed-rank	one median
Paired t	Sign, Wilcoxon signed-rank	paired differences
Two-sample t	Mann–Whitney U	two groups
One-way ANOVA	Kruskal–Wallis	k groups
Pearson correlation	Spearman correlation	association

What you're trading

Gains: no distributional assumption; robust to outliers; valid for ordinal data; often exact for small samples rather than relying on an approximation.

Costs:

Slightly less power when Normality really does hold (about 5% for the rank-based tests, 36% for the sign test).
Hypotheses are about medians or stochastic ordering, which are harder to interpret than means.
Confidence intervals and effect sizes are less convenient.
They are not assumption-free — Mann–Whitney still assumes independence, and its clean interpretation needs similar shapes.

A common misconception: nonparametric tests do not "fix" small samples. They remove the Normality assumption, but a tiny sample still has low power whatever test you run.

Worked example

Ten patients rate pain before and after treatment (0–10 scale, ordinal):

Before	8	7	9	6	8	7	9	8	6	7
After	5	6	6	5	4	7	6	5	5	4
Diff	3	1	3	1	4	0	3	3	1	3

Ordinal data on a small sample — a good case for nonparametrics.

Sign test. Drop the one zero, leaving n = 9. All nine remaining differences are positive, so S = 9.

$p = 2 \times P(S \ge 9 \mid n = 9, p = 0.5) = 2 \times (0.5)^9 = 0.0039$

Reject: pain decreased.

Wilcoxon signed-rank. Ranking the nine non-zero |d| values — five 3s, three 1s, one 4 — and summing the signed ranks gives W^+ = 45, the maximum possible, since every difference points the same way. p = 0.0039 by the exact distribution.

The two tests give exactly the same p-value here, and that is worth understanding. With every difference pointing the same way, both statistics are pinned at their ceilings — S = 9 out of 9, W^+ = 45 out of 45 — so each is asking "what is the chance of landing at the most extreme value possible?", and for both that is the same 2 \times (0.5)^9 = 0.0039.

The signed-rank test normally carries more information than the sign test: it uses the sizes of the differences, not just their directions. That extra information buys nothing in this sample, because there are no opposing signs for the magnitudes to weigh against. It pays off when the signs are mixed — a few large reductions against several tiny increases is a case the sign test calls a near-tie and the signed-rank test does not.

Effect size: the median reduction is 3 points on a 10-point scale, which is substantial regardless of the p-value.

Doing it in Python

The three main tests, on the pain data:

import numpy as np
from scipy.stats import wilcoxon, binomtest, ttest_rel

before = np.array([8, 7, 9, 6, 8, 7, 9, 8, 6, 7])
after = np.array([5, 6, 6, 5, 4, 7, 6, 5, 5, 4])
d = before - after

nonzero = d[d != 0]
positives = (nonzero > 0).sum()
sign_p = binomtest(positives, len(nonzero), 0.5).pvalue
print(f"sign test      : {positives}/{len(nonzero)} positive, p = {sign_p:.4f}")

# Drop the zero difference ourselves rather than leaving it to SciPy: the
# default zero-handling changed between SciPy versions, and with the zero
# removed the exact distribution is used, giving the same answer everywhere.
w = wilcoxon(nonzero)
print(f"signed-rank    : W = {w.statistic:.1f}, p = {w.pvalue:.4f}")

t = ttest_rel(before, after)
print(f"paired t (ref) : t = {t.statistic:.4f}, p = {t.pvalue:.6f}")

print(f"\nmedian reduction: {np.median(d)} points")

Robustness — what one outlier does to each test:

import numpy as np
from scipy.stats import ttest_1samp, wilcoxon

clean = np.array([1.2, 0.8, 1.5, 1.1, 0.9, 1.3, 1.0, 1.4, 0.7, 1.2])
dirty = np.append(clean, 50.0)             # one wild value

for name, data in [("clean", clean), ("with outlier", dirty)]:
    t = ttest_1samp(data, 0)
    w = wilcoxon(data)
    print(f"{name:>13}: t = {t.statistic:8.4f} (p {t.pvalue:.5f})   "
          f"W = {w.statistic:5.1f} (p {w.pvalue:.5f})")

print("\nThe t statistic collapses -- the outlier inflates s far more than xbar.")
print("The rank test barely notices: 50 is just 'the largest', same as 1.5 was.")

The efficiency trade-off, measured both ways:

import numpy as np
from scipy.stats import ttest_ind, mannwhitneyu

rng = np.random.default_rng(0)
n, trials, shift = 20, 3_000, 0.8

for name, sampler in [
    ("normal", lambda size: rng.normal(0, 1, size)),
    ("heavy-tailed (t3)", lambda size: rng.standard_t(3, size)),
]:
    t_hits = u_hits = 0
    for _ in range(trials):
        a = sampler(n)
        b = sampler(n) + shift
        t_hits += ttest_ind(a, b, equal_var=False).pvalue < 0.05
        u_hits += mannwhitneyu(a, b).pvalue < 0.05
    print(f"{name:>18}: t-test power {t_hits/trials:.4f}   Mann-Whitney {u_hits/trials:.4f}")

print("\nNormal data: the t-test wins slightly. Heavy tails: the rank test wins clearly.")

And a check that Mann–Whitney really counts pair comparisons:

import numpy as np
from scipy.stats import mannwhitneyu

rng = np.random.default_rng(1)
a = rng.normal(0, 1, 12)
b = rng.normal(0.5, 1, 15)

u = mannwhitneyu(a, b, alternative="two-sided")
by_hand = sum(1 for x in a for y in b if x > y) + 0.5 * sum(1 for x in a for y in b if x == y)

print(f"scipy U statistic : {u.statistic}")
print(f"pairs with x > y  : {by_hand}")
print(f"match: {u.statistic == by_hand}")
print(f"\nestimated P(X > Y) = {by_hand / (len(a) * len(b)):.4f}   (H0 says 0.5)")

Your turn

1. 12 paired differences: 8 positive, 3 negative, 1 zero. Run a sign test.

2. When would you prefer Mann–Whitney over a two-sample t-test?

3. Why is the sign test less powerful than the signed-rank test?

Solutions

1. Drop the zero, leaving n = 11 with S = 8 positives.

Under H_0, S \sim \text{Binomial}(11, 0.5). The two-sided p-value is

$p = 2 \times P(S \ge 8) = 2 \times \frac{\binom{11}{8} + \binom{11}{9} + \binom{11}{10} + \binom{11}{11}}{2^{11}} = 2 \times \frac{165 + 55 + 11 + 1}{2048} = 0.2266$

Fail to reject at \alpha = 0.05. Eight out of eleven sounds suggestive, but with this little data it's well within what chance produces.

2. Prefer Mann–Whitney when:

The data is ordinal — ranks, ratings, Likert items — where means aren't meaningful.
There are clear outliers or heavy tails that would distort \bar x and s.
The samples are small and visibly non-Normal, so the CLT can't rescue the t-test.
You want a conclusion that doesn't depend on assuming a distribution.

Prefer the t-test when the data is roughly Normal (it's more powerful, if only slightly) and when you want to talk about means, with an interpretable difference and confidence interval. "The groups differ by 4.2 units (95% CI 1.1–7.3)" is far more useful than "the ranks differ".

3. Because it uses strictly less information.

The sign test records only direction: differences of +0.1 and +50 are identical to it. The signed-rank test also uses the relative magnitudes, so a set of consistently large positive differences produces a more extreme statistic than a set of barely-positive ones.

Quantitatively, the ARE of the sign test against the t-test is $2/\pi \approx 0.64$ for Normal data, while the signed-rank test achieves $3/\pi \approx 0.955$. In sample-size terms the sign test needs about 57% more data for the same power.

The compensation is that the sign test assumes almost nothing — not even the symmetry the signed-rank test requires. When differences are wildly asymmetric, that extra robustness can be worth the lost power.

Check yourself in code

Run the sign test and the Wilcoxon signed-rank test on the pain data and confirm both reject at the 5% level.

Print exactly this:

positives 9 of 9
sign p 0.0039
wilcoxon p 0.0039
both reject: True

Round both p-values to 4 decimal places. Drop the zero differences before both tests — SciPy's default zero-handling for wilcoxon differs between versions, and removing the zero yourself gets the exact distribution (and the same answer) on any of them.

import numpy as np
from scipy.stats import wilcoxon, binomtest

before = np.array([8, 7, 9, 6, 8, 7, 9, 8, 6, 7])
after = np.array([5, 6, 6, 5, 4, 7, 6, 5, 5, 4])
d = before - after

nonzero = d[d != 0]
positives = int((nonzero > 0).sum())
print(f"positives {positives} of {len(nonzero)}")

# Run binomtest for the sign test and wilcoxon(nonzero) for the signed-rank
# test, print both p-values, then whether both are below 0.05.

import numpy as np
from scipy.stats import wilcoxon, binomtest

before = np.array([8, 7, 9, 6, 8, 7, 9, 8, 6, 7])
after = np.array([5, 6, 6, 5, 4, 7, 6, 5, 5, 4])
d = before - after

nonzero = d[d != 0]
positives = int((nonzero > 0).sum())
print(f"positives {positives} of {len(nonzero)}")

sign_p = binomtest(positives, len(nonzero), 0.5).pvalue
print("sign p", round(sign_p, 4))

w_p = wilcoxon(nonzero).pvalue
print("wilcoxon p", round(w_p, 4))
print("both reject:", bool(sign_p < 0.05 and w_p < 0.05))

Nonparametric tests replace distributional assumptions with signs and ranks. The sign test is maximally robust and least powerful; the Wilcoxon signed-rank and Mann–Whitney tests recover most of the lost efficiency by using rank magnitudes, at a cost of about 5% against the t-test under Normality — and they win outright when the tails are heavy.

That closes §5. Next: modelling one variable as a function of others.

39. Nonparametric tests