TL’DR: Bayesian A/B testing falls short
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
What is a posterior distribution?
The posterior distribution is what you believe about an effect after combining the prior with the data. It is a full distribution rather than a single estimate, and every Bayesian quantity a report shows is read off it.
In plain English
The posterior is the output of a Bayesian analysis and the object everything else is derived from. It assigns a probability density to every possible value of the effect: not "the lift is 3.2%" but a curve saying 3.2% is the most likely value, 1% and 5% are quite plausible, and 20% is nearly ruled out. Bayes' theorem produces it by combining the prior with the likelihood of the observed data.
Having a whole distribution rather than a point estimate is what makes the framework attractive to decision-makers, because it supports statements a frequentist analysis cannot make. The probability that the effect exceeds zero. The probability it exceeds the 2% threshold the business case needs. The expected loss from shipping the wrong arm. Each is a straightforward integral over the posterior, and each maps more directly onto a decision than "reject the null at 5%" does.
The corresponding frequentist quantities answer subtly different questions, and the difference is worth being precise about rather than glossing. A 95% credible interval contains the effect with 95% probability, given the prior and the data. A 95% confidence interval is generated by a procedure that captures the true effect 95% of the time across repeated studies. The first is the statement most people believe they are making when they report the second, which is the strongest practical argument for the Bayesian version.
What the posterior does not automatically deliver is safety under repeated inspection. It is often claimed that Bayesian methods are immune to the peeking problem, and this is only true of the posterior as a description of belief. If you monitor continuously and stop the moment the probability of being better crosses 95%, you will stop early on noise far more than 5% of the time — because that stopping rule selects favourable moments exactly as the frequentist version does. Anytime-valid Bayesian procedures exist and require deliberate construction.
The other practical caution is computational. Conjugate cases like Beta-Binomial give the posterior in closed form, which is why conversion-rate examples are always the ones in tutorials. Anything more complicated needs MCMC or variational approximation, and those bring their own diagnostics — convergence, effective sample size — which are easy to skip and produce confidently wrong posteriors when they fail.
The formula
One proportionality, and the three summaries that get reported from it.
- The posterior
p(θ | data) ∝ p(data | θ) · p(θ)Likelihood times prior, normalised. The normalising constant is what makes general cases computationally hard.
- Probability of being better
P( θ > 0 ) = ∫₀^∞ p(θ | data) dθAn integral over the posterior. The statement a p-value cannot make.
- Probability of clearing a threshold
P( θ > Δ )Substitute the effect the business case needs. Usually more decision-relevant than P(θ > 0).
- The conjugate shortcut
Beta(α, β) + Binomial data → Beta(α + s, β + n − s)Closed form for rates, which is why conversion examples dominate — see the proportion calculator.
Worked example
A conversion test with 30,000 users per arm gives control 1,440 conversions and the variant 1,566. The team fits a Beta-Binomial model with a weakly informative prior and reads several quantities off the resulting posterior on the relative lift.
A 98.8% chance the variant is better, and only an 80.1% chance it clears the threshold the business case actually requires.
The gap between those last two rows is what makes the posterior worth having. Almost any framework would call this a win — 98.8% is decisive, and the frequentist equivalent would be p = 0.012. But the decision on the table was not "is it better than zero", it was "does it clear 5%", and on that question the answer is a far less comfortable 80%. A team reading only the first number would forecast from an 8.7% lift with confidence; reading both, they know there is a one-in-five chance the change does not pay for itself. That reframing is available because the posterior describes the whole range rather than a verdict, and it is the strongest practical argument for the framework. Note the credible interval spans 1.4% to 16.3%, which is wide — 30,000 per arm is not much for a 4.8% base rate, and no choice of framework fixes that.
Common misconceptions
- “Bayesian methods let you monitor continuously without penalty.”
- The posterior is a valid description of belief at any moment, and a stopping rule that waits for it to cross a threshold still selects favourable moments. Stopping the first time P(better) exceeds 95% produces false positives well above 5%. Anytime-valid Bayesian procedures exist and have to be constructed deliberately.
- “A 95% credible interval and a 95% confidence interval say the same thing.”
- The credible interval contains the effect with 95% probability given the model. The confidence interval comes from a procedure that captures the truth 95% of the time across repeats — a statement about the method rather than about this interval. They often coincide numerically with a flat prior, and they are different claims.
- “The posterior gives you the answer without any assumptions.”
- It depends on the prior and on the likelihood model, both of which are assumptions. With plenty of data the prior fades and the model choice remains. Reporting a posterior without stating what produced it hides those choices behind a number that looks purely empirical.
Frequently asked questions
What should I report from a posterior?
The posterior mean or median, a 95% credible interval, and the probability of clearing whatever threshold the decision actually needs — not just the probability of being better than zero. That last distinction matters most: P(θ > 0) is usually high and rarely decision-relevant, while P(θ > business threshold) is the number someone can act on.
Can I look at the posterior as often as I like?
Looking is fine; stopping because of what you saw is not. The posterior is a valid statement of belief at every moment, and a rule that ends the experiment the first time it crosses a threshold selects favourable moments and inflates the error rate exactly as frequentist peeking does. Continuous monitoring needs a procedure designed for it, in either framework.
How is the posterior computed in practice?
For conversion rates, in closed form — a Beta prior with binomial data gives a Beta posterior, and updating is arithmetic. For anything more complex, by MCMC sampling or variational approximation, which return draws from the posterior rather than a formula. Those methods have diagnostics worth respecting: unconverged chains or a low effective sample size produce a posterior that looks fine and is not.
Further reading
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.