What is the peeking problem?

A fixed-horizon significance test promises that if nothing is going on, it will falsely claim an effect 5% of the time — provided you look once, at the sample size you planned for. Look every day and stop the moment something crosses 0.05, and that promise is void. Over a fortnight of daily checks the false-positive rate rises to roughly 25%, so one experiment in four reports a win on a change that does nothing.

The mechanism is easier to see as a random walk. Under the null the observed difference wanders around zero, and the p-value wanders with it. On any single day it is unlikely to be below 0.05 — but across fourteen days there are fourteen chances for the walk to stray far enough, and stopping at the first excursion selects exactly those runs. Given unlimited time and unlimited checking, a fixed-horizon test will cross any threshold eventually with probability one.

The crucial distinction is between looking and acting. Monitoring a running experiment for operational problems is harmless and necessary — checking for a sample ratio mismatch, watching error rates, confirming data is arriving. What inflates the error rate is letting the observed result determine when to stop. Peeking is a stopping-rule problem, not a viewing problem, and a team that looks daily and always runs to its pre-registered sample size has not peeked in the sense that matters.

There are two legitimate ways to stop early. Alpha spending allocates the error budget across a fixed number of pre-planned interim analyses, so each look uses a stricter threshold and the total stays at 5%. Always-valid inference uses statistics that remain valid under continuous monitoring, letting you check as often as you like at the price of wider intervals. Both work; neither is free, and choosing one before the test starts is what makes them valid.

The organisational reality is that peeking is rarely a deliberate statistical choice. It happens because a dashboard shows a green result on day three and stopping feels like efficiency rather than a decision about error rates. That is why the defence has to be structural — declare the duration with the sample size calculation, and either hide interim estimates or adopt a method that survives being looked at.

A simulation runs 10,000 A/A experiments — no real effect — each accumulating 40,000 users per arm over 20 days. Three stopping rules are compared: look once at day 20, look daily and stop at the first p below 0.05, and look daily under an alpha-spending boundary.

The middle row is the whole problem: more than a quarter of experiments on changes that do nothing would have been shipped as wins. The median stopping day of 6 explains why it feels harmless — the result appeared early and looked decisive, which is exactly what a lucky excursion looks like from the inside. The alpha-spending row shows the cost of doing it properly: it also permits stopping early, and its median stop is day 17 rather than day 6, because the early boundaries are deliberately strict. That is the trade. You can monitor continuously and stop early, and you cannot do so at the same threshold you would use for a single look. Note also what the first row establishes — the fixed-horizon test is correctly calibrated at 4.9%, so nothing is wrong with the underlying statistics. The failure is entirely in the stopping rule.