What is Bayesian A/B testing?

Bayesian A/B testing replaces the reject-or-not verdict with a distribution. Instead of "p = 0.03, significant", the output is a posterior over the effect, from which you read the probability the variant is better, the probability it clears a business threshold, and the expected loss from choosing wrongly. Those outputs map more directly onto a decision than a p-value does, and that is the honest case for it.

The genuine advantages are worth stating plainly. The outputs are interpretable — "92% chance the variant is better" means what a reader thinks it means, whereas a p-value almost never does. Decision-theoretic framing follows naturally, so a threshold can be set on expected cost rather than on an arbitrary 5%. And where a real prior exists, it is used rather than discarded, which is a genuine efficiency gain that frequentist analysis leaves on the table.

The reason most teams actually adopt it is a misconception, and it is worth being direct about. Bayesian methods are widely believed to remove the peeking problem — check whenever you like, stop when the probability of being better crosses 95%. That is false. A posterior is a valid statement of belief at every moment, and a stopping rule that waits for it to cross a threshold selects favourable moments exactly as the frequentist version does. Simulations of that rule produce error rates well above the nominal level. Anytime-valid Bayesian procedures exist; the default implementation in most tools is not one.

The second common disappointment is that the prior rarely does much work. Most platforms default to something flat or near-flat, because a per-experiment informative prior requires history nobody has assembled — and with a flat prior on a decent sample the Bayesian answer is a re-expression of the frequentist one. Teams switch expecting a more powerful method and receive the same evidence in friendlier language, which is a real gain in communication and not the gain they were sold.

Where it genuinely earns its place is where the prior carries information: marketing mix modelling, where domain knowledge about channel behaviour and adstock is substantial and hard-won, or small-sample settings where past effect sizes are a better guide than a thin experiment. For a high-traffic conversion test with no informative prior, the choice of framework is mostly a choice about how to phrase the answer.

A simulation runs 10,000 A/A experiments — no real effect — under a Bayesian analysis with a flat prior, comparing two stopping rules: run to a fixed 40,000 users per arm, or check daily and stop when P(variant better) first exceeds 95%.

The two failure rates being 24.7% and 26.3% is the point. Switching framework did almost nothing to the error rate, because the problem was never the framework — it was the stopping rule, which selects favourable moments regardless of what statistic is being watched. The 0.98 correlation in the last row says the same thing from another angle: with a flat prior these two analyses are near-identical instruments wearing different labels. The first row is the reassuring one: run to a fixed horizon and the Bayesian analysis is correctly calibrated at 5.1%. So the practical guidance is the same in both frameworks — fix the duration in advance, or adopt a procedure explicitly built for continuous monitoring. What Bayesian analysis genuinely offers is better-phrased output, and it is worth having for that reason alone rather than for a peeking immunity it does not possess.