Oops I Did It Again: Common Testing Mistakes and How to Fix Them
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
What is p-hacking?
P-hacking is exploiting the flexibility in how data is analysed until a result crosses the significance threshold. It rarely involves anyone deciding to cheat — the same outcome arrives through a series of individually defensible choices made after seeing the data.
In plain English
Every analysis involves choices. Which metric is primary, which users to exclude, whether to cap outliers and at what threshold, which segments to examine, when to stop collecting. Make those choices after seeing the data and you can search — deliberately or not — until something reaches significance. That search is p-hacking, and the resulting p-value describes a procedure nobody performed.
The framing that makes it recognisable rather than accusatory is the garden of forking paths. Most p-hacking involves no dishonesty at all: an analyst hits a flat primary metric, notices that excluding bot-like sessions is defensible, tries it, sees the result improve, and keeps it. Each decision is reasonable in isolation, and each was made with knowledge of what it did to the answer. The false-positive rate is set by all the paths that could have been taken, not by the one that was.
In experimentation the flexibility concentrates in a few places. Stopping when the result looks good is the peeking problem. Reading fifteen metrics and reporting the significant one is multiple comparisons without a correction. Slicing by device, country, tenure and channel until a segment shows an effect is the same thing with more paths. Choosing an outlier cap after seeing how it changes the result is a subtler version, and the changing definition of a metric mid-analysis is subtler still.
The defence is pre-registration, and it need not be elaborate. Write down the primary metric, the sample size, the duration, the exclusion rules and the segments of interest before the experiment starts, and treat everything else as exploratory. It costs ten minutes and it converts a garden of forking paths into a single path. What makes it work is not formality but timing: the decisions must be made when they cannot be influenced by their effect on the answer.
Exploratory analysis remains valuable and should not be discouraged — most good hypotheses come from looking at data without a plan. What matters is labelling it honestly. A segment that looks interesting is a candidate for the next experiment, not a finding from this one, and an organisation that keeps that distinction can explore freely without accumulating beliefs that no experiment established.
The formula
The arithmetic of how quickly flexibility manufactures significance, which is faster than most people expect.
- The paths multiply
paths = metrics × segments × exclusion rules × stopping pointsThree metrics, four segments and two exclusion rules is 24 analyses, not one.
- Chance of at least one significant result
1 − ( 1 − α )^kk = 24 at α = 0.05 gives 71%. Even correlated paths, which reduce this, leave it well above 5%.
- What the reported p-value describes
a single pre-specified testNot the best of 24. Nothing in the output records how many paths were explored.
- The fix
fix metric, sample size, exclusions and segments before launchConverts the garden into one path — see the sample size calculator for the duration half.
Worked example
An experiment on a genuinely inert change comes back flat. An analyst, acting in good faith, works through a sequence of reasonable adjustments. Each step is simulated 10,000 times over null data to see how often it eventually yields a significant result.
Six layers of defensible flexibility take the false-positive rate from 5% to 78% on data with no effect in it whatsoever.
No step in that list is disreputable. Checking secondary metrics is diligent. Capping outliers is often correct. Excluding three-second sessions removes probable bots. Segmenting by device is standard practice. Stopping when the answer is clear is efficient. The problem is that all six were performed with visibility of what each did to the p-value, and the reported number describes only the last one. The rate would be lower in practice because these paths are correlated rather than independent — but the direction and the order of magnitude are right, and 78% is not far from what a genuinely determined search achieves. The remedy is not to forbid any of these operations; it is to decide them in advance. An outlier cap chosen from historical data before launch is a good analysis; the same cap chosen because it produced p = 0.04 is not, and the two are indistinguishable in the output.
Common misconceptions
- “P-hacking means deliberately manipulating results.”
- It rarely involves any intent to deceive. The usual route is a sequence of individually defensible choices made with knowledge of their effect on the answer — the garden of forking paths. Treating it as a question of honesty makes it harder to address, because most people doing it are quite correctly confident that they were not cheating.
- “Exploring the data thoroughly is good practice, not p-hacking.”
- Exploration is valuable and the problem is what it is called afterwards. A finding discovered by searching is a hypothesis; presenting it with a p-value from the same data treats a search as a test. Explore freely, label it as exploratory, and confirm anything that matters in a new experiment.
- “Correcting for multiple comparisons solves it.”
- It addresses one route — several metrics tested at once — and leaves the rest. Flexible exclusions, outlier thresholds chosen after the fact, post-hoc segmentation and opportunistic stopping all remain, and correcting for the paths you happen to remember taking is not a correction. Pre-specification is what closes the whole class.
Frequently asked questions
What does pre-registration need to contain?
The primary metric, the sample size and duration, the exclusion and outlier rules, and any segments you intend to examine. That is enough, and it takes about ten minutes to write. An internal document attached to the experiment record is sufficient — the point is that the decisions exist before the data can influence them, not that anyone else validates them.
How do I explore data without p-hacking?
Explore as much as you like and report it as exploratory, with effect sizes and intervals rather than significance claims. Anything that looks important becomes the pre-registered primary metric of a follow-up experiment, which either confirms it or does not. The distinction that matters is between generating a hypothesis and testing one, and the same dataset cannot do both.
Can you detect p-hacking in someone else's analysis?
Not reliably in a single result, since the analysis path is not recorded. Across many results it shows up as a cluster of p-values just below 0.05 where a smooth distribution would be expected. In practice the useful questions are procedural: was the metric declared in advance, was the duration fixed, were the exclusions decided before the data arrived. A result that cannot answer those is not necessarily wrong, and it has not been tested in the way its p-value implies.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.