Bell Statistics

Validity threats and corrections

The ways a technically correct test produces a wrong answer — peeking, multiplicity, broken randomisation — and the corrections that address each one.

14 terms

Every term in this group describes a way of getting a result that is arithmetically correct and substantively false. That is what makes them worth their own category: none of them is a mistake in the calculation, so none of them is caught by checking the calculation. They are failures of the procedure around it.

The two that account for most false positives in practice are peeking and multiplicity, and they are the same error wearing different clothes. Checking a test daily and stopping at the first significant reading turns a 5% false-positive rate into roughly 25% over a fortnight. Testing twelve metrics at 5% gives about a 46% chance that at least one clears the bar with nothing happening at all. In both cases the fix is to decide the rule before seeing the data — see multiple comparisons and sequential testing.

The third kind is the test that was broken before it produced a number. A sample ratio mismatch says the two arms were not built the same way, and once that is true no correction to the analysis rescues it — the data describe two populations that were never comparable. Diagnosing it is the first thing to do with a surprising result and the last thing most teams reach for.

Terms in this group

  • Alpha spending

    Divide the 5% across your planned looks — strict early, lenient at the end, and the schedule is fixed before you start.

  • Always-valid inference

    Statistics that stay honest under continuous monitoring — you may stop whenever you like, and it costs about a third more traffic.

  • Bonferroni correction

    Divide alpha by the number of tests — bulletproof, arithmetic you can do in your head, and strictly beaten by Holm.

  • False discovery rate

    Bound the share of your wins that are wrong rather than the chance of any error — the right trade on twenty metrics.

  • Family-wise error rate

    The chance of at least one false alarm across the whole set — 5% per test becomes 46% across twelve.

  • Multiple comparisons

    Test enough things and something will look significant — the arithmetic, and the four fixes.

  • P-hacking

    Enough defensible choices, made after seeing the data, will find significance in anything.

  • Peeking problem

    Look often enough and something crosses the line — checking daily for a fortnight turns 5% into roughly 25%.

  • Sample ratio mismatch

    The split is wrong, so the randomisation is broken — the cheapest and most decisive check you can run.

  • Sequential testing

    How to look at a running test without breaking it — and what peeking costs when you do not.

  • Simpson's paradox

    Wins in every segment, loses overall — and the aggregate is the one you should distrust.

  • Survivorship bias

    Measuring only what remains — and the departures are usually the ones that would have changed the answer.

  • Twyman's law

    Any figure that looks interesting is usually wrong — and the bigger the result, the cheaper it is to check.

  • Winner's curse

    The lift you measured is not the lift you will get — selection inflates it, and marginal winners inflate it most.

Other topics

Back to the glossary