Choosing a test

Which significance test the data in front of you calls for, what each one assumes, and what breaks when the assumption does not hold.

16 terms

Almost every test in this group answers the same question — is the difference between these groups bigger than noise would produce on its own — and they differ only in what the data look like. Picking wrongly rarely produces an error message. It produces a p-value that is confidently wrong in a direction you cannot see.

The first fork is the kind of outcome. A binary one — converted or did not — is a proportion, compared with a z-test or, laid out as a table, a chi-square test. A measured one, such as revenue or session length, is a mean, compared with a t-test. Three or more groups at once is ANOVA, which exists to avoid running every pair separately and inflating the false-positive rate.

The second fork is whether the observations are independent. Different people in every row, and they are. The same people measured twice, and they are not — a paired t-test is then both the correct answer and a far more sensitive one, because it removes the differences between people from the comparison entirely.

The third is whether the distribution behaves. Revenue is skewed and regularly dominated by a handful of large customers, which is exactly where a mean is a poor summary and the Mann-Whitney U test compares by rank instead. Each entry says what it assumes and which calculator computes it.

ANOVA
One test for three or more variants — and the reason it deliberately refuses to say which one won.
Bootstrap
Resample your own data ten thousand times and watch the answer wobble — uncertainty for statistics with no formula.
Chi-square test
The test for counts in a table — the standard SRM check, and the reason expected counts matter more than sample size.
Degrees of freedom
The count of what is still free to move — and the number that decides how demanding your significance threshold is.
Equivalence test
The only test that can conclude "these are the same" — done as two one-sided tests against a band you set first.
Fisher's exact test
Counts the possible tables instead of approximating them — the right test when a cell expects fewer than five.
Interaction effect
When two changes stop being independent — the reason concurrent tests can both read as wins and lose money together.
Log-rank test
For questions about when rather than whether — and it uses the people who have not converted yet instead of discarding them.
Mann-Whitney U test
Compares by rank instead of by mean, so one whale cannot move the result — and answers a subtly different question.
McNemar's test
The paired test for yes-or-no outcomes — and it throws away every subject who did not change their mind.
Non-inferiority test
Not "is it better" but "is it not meaningfully worse" — and the margin you choose decides the entire answer.
One-tailed vs two-tailed test
The choice that halves your p-value and costs you the ability to see damage — and why it must be made before the data arrives.
Paired t-test
When every observation has a partner, pairing removes the differences between units and can cut the sample you need by an order of magnitude.
T-distribution
The bell curve's cautious cousin — and the reason a small sample needs a bigger result to clear the same bar.
T-test
The workhorse for comparing two averages — and the four assumptions that decide whether its answer means anything.
Z-test
The test behind every conversion-rate readout — and the one condition that decides whether it is safe to use.

A/B Testing at Bell Statistics

We build the analysis layer that picks the right test automatically, so the readout is correct without anyone having to remember which case they are in. See how we work.

Choosing a test

Terms in this group

A/B Testing at Bell Statistics

Other topics