TL’DR: Bayesian A/B testing falls short
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
What is Bayes' theorem?
Bayes' theorem describes how to update a belief when new evidence arrives, combining what you thought beforehand with how likely the evidence is under each possibility. Its practical lesson is that the base rate matters as much as the evidence.
In plain English
Bayes' theorem answers a question that arises constantly and is answered wrongly almost as often: given some evidence, how much should I believe a hypothesis? Its structure says the answer depends on two things, not one. How likely the evidence is if the hypothesis is true, and how likely the hypothesis was before any evidence arrived. Leaving out the second is the single most common reasoning error in applied statistics.
The canonical illustration is diagnostic. A test is 95% accurate for a condition that affects one person in a thousand. Someone tests positive; the intuitive conclusion is a 95% chance they have it. The correct answer is under 2%. Out of 100,000 people, 100 have the condition and 95 test positive; 99,900 do not and about 4,995 test positive anyway. The positives are dominated by false ones because the condition is rare, and no amount of test accuracy overcomes a base rate that extreme.
The same structure explains why a p-value is not the probability your result is wrong. A p-value is the probability of the evidence given the null — one half of the theorem. Converting it into a probability about the hypothesis requires the prior, which the frequentist calculation never touches. In a mature product where most tested ideas do nothing, a result at p = 0.05 can still leave a substantial chance that nothing is there, and that is arithmetic rather than opinion.
In the Bayesian framework the theorem is the engine rather than an aside. A prior distribution encodes the belief before the experiment, the likelihood describes what the data says, and the posterior is the updated belief the theorem produces. Everything else in this group — credible intervals, probability to be best, expected loss — is read off that posterior.
The practical use in experimentation does not require adopting the framework wholesale. Simply asking "how plausible was this before we tested it, and how many ideas like it have worked?" is applying the theorem informally, and it is what stops a surprising result from being taken at face value. It is the reasoning behind Twyman's law, stated formally.
The formula
One identity, and the odds form that makes the base-rate lesson immediate.
- The theorem
P(H | E) = P(E | H) · P(H) / P(E)Posterior equals likelihood times prior, normalised. P(H) is the base rate that intuition drops.
- The odds form
posterior odds = prior odds × likelihood ratioOften easier to reason with: evidence multiplies the odds you started with rather than replacing them.
- The diagnostic example
0.95 × 0.001 / ( 0.95 × 0.001 + 0.05 × 0.999 ) = 0.0187A 95% accurate test on a 1-in-1000 condition gives a 1.9% chance of actually having it.
- What a p-value supplies
P( E | H₀ ), not P( H₀ | E )One half of the theorem. The other half is the prior, which the calculation never sees — see p-value.
Worked example
A product team has run 200 experiments over two years. Reviewing them honestly, about 20% produced a real effect and the rest did nothing. A new test comes back at p = 0.04. The question is how much to believe it, using the programme's own history as the prior.
A significant result from this programme has an 80% chance of reflecting a real effect — not the 96% the p-value invites.
80% is a respectable number and it is not what most readers take from p = 0.04. The gap comes entirely from the base rate: four in five ideas do nothing, so even a well-run test produces a meaningful minority of false positives among its wins. Two things change that figure sharply, and both are under the team's control. Raising power raises the true-positive count — at 40% power the same arithmetic gives 8 true against 4 false, so only 67% of significant results would be real. And a programme testing better-founded ideas has a higher prior and therefore more trustworthy wins. That is the useful practical reading: the reliability of your significant results depends on the quality of your hypotheses and the power of your tests, not only on the threshold. Note the prior here came from the programme's own history rather than being invented, which is what makes the calculation defensible rather than rhetorical.
Common misconceptions
- “A p-value of 0.04 means there is a 4% chance the result is wrong.”
- It is the probability of seeing data this extreme if there were no effect — one half of Bayes' theorem. Turning it into a probability about the hypothesis requires the prior, and where most tested ideas do nothing, a result at p = 0.04 can leave a 20% or greater chance of being a false positive.
- “Bayes' theorem requires you to make up a subjective prior.”
- Priors can be estimated from data. A programme's historical hit rate is an empirical prior, as is the distribution of effect sizes from past experiments. Subjectivity enters when no such history exists, and even then stating an assumption explicitly is more transparent than an analysis that assumes something implicitly.
- “With enough data the prior stops mattering.”
- True in the limit and frequently irrelevant in practice. Where the evidence is strong the posterior is dominated by the data; where the base rate is extreme — rare conditions, rarely-successful interventions — the prior continues to dominate at any realistic sample size. The diagnostic example does not improve with a larger population.
Frequently asked questions
What is the base rate fallacy?
Judging how likely a hypothesis is from the strength of the evidence alone, ignoring how common the hypothesis was to begin with. It is why a 95% accurate test for a rare condition mostly returns false positives, and why a significant result on an implausible idea is more likely to be noise than a discovery. The correction is always the same: ask how many cases like this are real before asking how convincing this one looks.
How does Bayes' theorem apply to A/B testing?
Most usefully as a check on interpretation rather than as a change of method. Knowing your programme's historical hit rate lets you convert a p-value into a rough probability that a given win is real, which is frequently sobering. Adopting the framework fully means priors, posteriors and credible intervals — a bigger change, and one worth making only where the prior carries genuine information.
Where do I get a prior from?
From your own history where possible: the share of past experiments that produced real effects, and the distribution of effect sizes among those that did. That makes the prior an empirical quantity rather than an opinion. Where no history exists, use a deliberately wide prior and report how sensitive the conclusion is to it — a result that flips when the prior changes is telling you the data is not doing the work.
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.