Validity threats and corrections
The ways a technically correct test produces a wrong answer — peeking, multiplicity, broken randomisation — and the corrections that address each one.
Every term in this group describes a way of getting a result that is arithmetically correct and substantively false. That is what makes them worth their own category: none of them is a mistake in the calculation, so none of them is caught by checking the calculation. They are failures of the procedure around it.
The two that account for most false positives in practice are peeking and multiplicity, and they are the same error wearing different clothes. Checking a test daily and stopping at the first significant reading turns a 5% false-positive rate into roughly 25% over a fortnight. Testing twelve metrics at 5% gives about a 46% chance that at least one clears the bar with nothing happening at all. In both cases the fix is to decide the rule before seeing the data — see multiple comparisons and sequential testing.
The third kind is the test that was broken before it produced a number. A sample ratio mismatch says the two arms were not built the same way, and once that is true no correction to the analysis rescues it — the data describe two populations that were never comparable. Diagnosing it is the first thing to do with a surprising result and the last thing most teams reach for.