ANOVA compares three or more group means at once by asking whether the variation between the groups is larger than the variation within them. It answers whether any of the groups differ, in a single test, without comparing every pair separately.
FRun an A/B/C/D test and the obvious move is six pairwise t-tests, one for each pair. The obvious move is also a trap: six tests at the 5% level give roughly a 26% chance that at least one comes back significant when all four variants are identical. ANOVA exists to ask the question once. It returns a single p-value for "do any of these groups differ", spending 5% in total rather than 5% per comparison.
The name describes the mechanism, which is less intuitive than the goal. To find out whether means differ, ANOVA compares two kinds of variation. Between-group variation measures how far the group means sit from the overall mean; within-group variation measures how much individual observations scatter around their own group's mean. If the groups are genuinely alike, both quantities estimate the same underlying noise and their ratio should sit near 1. If the groups differ, the between-group term is inflated by real differences and the ratio grows. That ratio is the F statistic.
The critical limitation is that a significant F tells you something is going on and refuses to say what. With four variants and p = 0.008 you know they are not all the same and you do not know whether C beat A, D beat everything, or B is dragging the rest down. Finding out requires follow-up comparisons — and those bring back exactly the multiplicity problem ANOVA sidestepped, so they need a correction such as Tukey's or Bonferroni. ANOVA is a gatekeeper rather than an answer.
For most online experiments there is a better framing available, and it is worth saying plainly. A four-arm test usually has a control and three challengers, and the question is not "are these four different" but "does any challenger beat control". That is three comparisons against a baseline, not six among peers, and Dunnett's correction handles it with more power than an omnibus F followed by pairwise cleanup. ANOVA earns its place when the groups are genuinely symmetric — four pricing tiers, five onboarding variants with no incumbent.
Its assumptions are the t-test's, extended: independent observations, roughly normal sampling distributions, and — for the classical version — similar variances across groups. That last one is the fragile one, and Welch's ANOVA relaxes it the same way Welch's t-test does. When the outcome is badly skewed, the Kruskal-Wallis test is the rank-based counterpart, standing in the same relation to ANOVA as the Mann-Whitney U test does to the t-test.
One ratio of two variance estimates. The whole method is deciding how much of the total scatter belongs between the groups and how much belongs inside them.
F = ( between-group variance ) / ( within-group variance )Near 1 when the groups are alike. Large when the group means are further apart than internal scatter explains.
MS_between = Σ nᵢ ( x̄ᵢ − x̄ )² / ( k − 1 )k groups. Each group's distance from the grand mean, weighted by its size.
MS_within = Σ Σ ( xᵢⱼ − x̄ᵢ )² / ( N − k )The pooled noise estimate. N observations in total — see the one-way ANOVA calculator.
df_between = k − 1, df_within = N − kThe F distribution takes both, which is why an F is always quoted as F(df₁, df₂) — see degrees of freedom.
F = t²With k = 2 this is algebraically a t-test. ANOVA is the generalisation rather than a different idea.
A subscription business tests four onboarding flows on day-30 retention, measured as sessions in the first month. Each arm has 2,500 users. The means are 8.1, 8.4, 9.2 and 8.3 sessions, with a pooled within-group standard deviation of 4.6. Six pairwise t-tests were the alternative under consideration.
p < 0.0001. The four flows are not interchangeable. Which one to ship is a question this test does not answer.
The F is decisive and it is also where the analysis begins rather than ends. Flow C is a full session ahead of A, which looks like the story, but the omnibus test has not established that C beats A — only that the four means are further apart than noise explains, which flow B being unusually low would also produce. Tukey's HSD on the six pairs confirms C against each of the others and finds no separation among A, B and D. Worth noticing what six uncorrected t-tests would have cost here: a 26% chance of at least one spurious significant pair even if all four flows were identical, and with a real effect present that noise sits underneath a true finding and makes it hard to tell which pairs to believe. Also worth noticing that if A were the incumbent, Dunnett's against control would have been the sharper design from the start.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
In this post, we will dive into the various methods for addressing multiple comparisons, examining their strengths and weaknesses, and providing guidance on when each method is most appropriate to use.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.