Bayesian methods

Reasoning about effects as probability distributions rather than reject-or-not decisions: what that buys, what it costs, and what it does not fix.

7 terms

Frequentist testing asks whether the data would be surprising if nothing were going on. Bayesian methods ask a different question — given what we believed beforehand and what we have now seen, what should we believe about the effect? The answer is a whole distribution rather than a verdict, which supports statements like "an 88% chance the variant is better" that a p-value cannot make.

That reframing is genuinely useful and it is routinely oversold. A posterior distribution makes decision-theoretic reasoning natural: expected loss puts a cost on shipping the wrong arm, which is closer to the actual business question than a significance threshold. Where it is oversold is as a remedy for the peeking problem — Bayesian quantities are not automatically safe under continuous monitoring, and treating them as though they were reintroduces the error it was meant to remove.

Everything turns on the prior. Where real information exists — historical effect sizes, a well-understood market, the domain knowledge that goes into marketing mix modelling — a prior is an asset, and Bayesian methods are the natural framing. Where it does not, an uninformative prior mostly reproduces the frequentist answer in different notation, at the cost of a decision nobody examined.

These entries say what each quantity means, how to read it honestly, and where the framework earns its place against the alternative.