What is a prior distribution?

Every Bayesian analysis starts with a prior: a probability distribution over the effect, representing what was believed before the experiment. Bayes' theorem combines it with the data to produce the posterior, and everything reported afterwards depends on both. The prior is the part that has no frequentist equivalent, and it is where most of the argument about Bayesian methods lives.

Priors range from informative to deliberately vague. An informative prior says something specific — past experiments on this surface produced effects between −2% and +5%, centred near zero — and it can be built from a programme's own history rather than invented. A weakly informative prior rules out the absurd while staying open: effects beyond ±50% are implausible for a button colour. A flat or uninformative prior tries to say nothing at all, and mostly reproduces the frequentist answer in different notation.

The honest case for an informative prior is that it is usually true and usually ignored. If ninety of your last hundred experiments produced effects under 3%, then a claimed 40% lift should be discounted, and a prior does that automatically and transparently. A frequentist analysis discards that history entirely and treats each experiment as though nothing were known. Where the history is real, the prior is an asset rather than a contamination.

The dishonest case is a prior chosen because it produces the desired conclusion. A prior centred on a large positive effect will report a large positive effect from thin data, and nothing in the output announces that the conclusion came from the assumption. This is why the prior must be stated, justified from something external, and fixed before the data arrives — the same discipline that makes a pre-registered hypothesis meaningful.

The practical safeguard is a sensitivity check. Rerun the analysis under two or three defensible priors — a flat one, an empirical one from history, a sceptical one centred at zero — and see whether the conclusion holds. If it does, the data is doing the work and the prior is a detail. If it flips, the prior is doing the work, which is worth knowing and worth saying out loud rather than reporting one number.

A conversion test on a 5% baseline observes 62 conversions in 1,000 users for the variant against 50 in 1,000 for control. The analysis is run under three priors to see how much the choice matters at this sample size.

The spread across those three numbers is the finding, and it is what a single reported figure would have concealed. At 1,000 users per arm the data is thin enough that the prior still carries real weight — the flat prior lets the observed 24% relative lift speak for itself, while the empirical prior correctly notes that lifts that large are rare in this programme and pulls the estimate back. The empirical version is the defensible one here, because it comes from 200 actual experiments rather than from a default. Worth noticing the last row: the frequentist p-value of 0.226 would have been reported as "no significant difference", which is a third framing again. None of these is wrong; they answer different questions. What would be wrong is running all three and reporting whichever sounded best, which is why the prior has to be fixed before the data arrives.