Always-valid inference produces p-values and confidence sequences that remain correct no matter how often they are checked or when the experiment is stopped. It removes the peeking problem by construction, at the cost of needing more data to reach the same conclusion.
A conventional p-value is valid at one moment: the sample size you planned for. Check it repeatedly and stop when it looks good and it stops meaning what it claims — that is the peeking problem. Always-valid inference is the family of methods that removes the constraint entirely. The p-value is constructed so that its guarantee holds at every point simultaneously, so any stopping rule at all is legitimate, including one that depends on the data.
The corresponding interval is a confidence sequence rather than a confidence interval: a sequence of intervals with the property that the true effect lies inside all of them simultaneously with 95% probability. That simultaneity is the whole point and it is a strictly stronger claim than an ordinary interval makes. It is also what lets a dashboard show a live estimate that nobody can misuse by looking at the wrong moment.
The best-known implementation is mSPRT — the mixture sequential probability ratio test — which extends Wald's sequential test by integrating over a prior distribution on the effect size rather than testing a single fixed alternative. That mixture is what makes it work without knowing the effect in advance, and the prior's width is a real parameter: a wider one is more robust to being wrong about the plausible effect and costs more data, a narrower one is sharper if the guess is good.
The cost is a sample premium, typically 20 to 40% more traffic to reach the same conclusion as a fixed-horizon test that ran to completion. That is the price of the anytime guarantee and it is not a defect. Compared with the alternative — a fixed test that gets peeked at anyway — it is clearly worth paying. Compared with a fixed test that is genuinely run to its horizon, it is a real cost, and the choice depends on whether your organisation can actually leave experiments alone.
The related option is alpha spending, which permits a fixed number of pre-planned interim looks rather than unlimited ones. It is more efficient than always-valid inference for the looks it allows, and more rigid: the schedule must be set in advance and adhered to. Always-valid inference is the better fit for a platform with live dashboards where anyone can look at any time; alpha spending suits a small number of planned checkpoints.
The guarantee is the definition. The mSPRT statistic below is the standard way of achieving it.
P( ∃ t : p_t ≤ α | H₀ ) ≤ αThe probability of EVER crossing the threshold is bounded, not just the probability at one fixed time.
Λ_t = ∫ ( likelihood under θ / likelihood under H₀ ) dπ(θ)A likelihood ratio mixed over a prior π on the effect. The mixture is what removes the need to name the alternative.
p_t = min( 1, 1 / Λ_t ), monotone decreasing in evidenceNever increases with more data, which is what makes stopping at any point safe.
n_always-valid ≈ 1.2 to 1.4 × n_fixedThe price of the guarantee — see the sample size calculator for the fixed-horizon baseline.
A team compares three approaches on a 4% conversion metric where they want to detect a 5% relative lift. Their organisational reality is that experiments get checked daily and stopped when they look decisive, which has happened on most recent tests.
The premium is 31% in the worst case and the median real effect is detected in 15 days rather than 21, with the error rate held at 5%.
The two middle rows are what make the case here, and they pull in opposite directions. Always-valid inference does cost 31% more traffic if you run it to the full horizon — that is the honest headline premium. But its whole point is that you do not have to: when a real effect exists it is typically detected well before that, and the median stopping point of 94,000 per arm is actually less than the fixed test's 128,000. The premium is paid only in the cases where the effect is small or absent, which are exactly the cases you want to run longer anyway. Against this team's actual practice rather than their intended practice, it is not a cost at all — they were already stopping early at a 26% error rate, and always-valid inference gives them the same convenience at 5%. That comparison is the one to make: not against the fixed test you designed, but against the fixed test you run.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.
