One-way ANOVA calculator

Use one-way ANOVA when you have one continuous outcome and three or more independent groups, and the first question is whether the means differ at all. Four onboarding variants, five pricing pages, three routing rules — each unit belongs to exactly one group and no unit appears twice. With two groups it reduces exactly to the two-sample t-test (F = t², same p-value), so there is no reason to use it there.

The reason it exists is error rate, not convenience. Six pairwise t-tests across four arms at α = 0.05 give you roughly a one-in-four chance of a false positive when every arm is identical, and that arithmetic turns a multi-arm test into a machine for imaginary winners. The omnibus F holds the family-wise rate at α for the one question it asks. What it does not do is answer the question you have: "at least one of these four differs from at least one other" is almost never a decision. Getting from there to "variant C beats control" is a separate inferential problem with its own correction — see choosing a multiple-comparison correction and why the problem is worse than it looks.

Use something else when the data says so. Conversion outcomes across several arms belong in a chi-square test; time-to-event outcomes want the log-rank test. If the same units are measured under every condition the groups are not independent and this model is wrong — you want a repeated-measures design, or the paired t-test for two conditions. For badly behaved continuous data the rank analogue is Kruskal–Wallis, the k-group generalisation of the Wilcoxon rank-sum test, with the same caveats about what it does and does not compare.

ANOVA inherits the t-test's assumptions and adds one about variances. They are not equally important, and the received order of importance is roughly backwards:

• Independence, within and across groups. The one that matters and the one no diagnostic reveals. Clustered assignment — by account, by geography, by session rather than by user — inflates F in a way that looks exactly like a real effect, and randomisation done properly is the only defence.
• Normality of the residuals, not of the outcome. With a few hundred observations per arm this is close to a non-issue; with a dozen per arm and a long right tail it is not, because F is a ratio of variances and variances are far more sensitive to extreme values than means.
• Equal variances across groups — and unlike Welch's t-test, the classical F does not drop this one. With balanced arms it is fairly robust. With unbalanced arms it breaks in both directions: anti-conservative when the smaller groups carry the larger variances, over-conservative when they carry the smaller. Unbalanced and heteroscedastic, use Welch's ANOVA.
• No hidden multiplicity in the metric. ANOVA controls error across groups, not across outcomes: running it on nine metrics and reporting the significant one reintroduces the exact problem the F test prevents — see common testing mistakes.
• And, unstated but assumed, that the omnibus null is worth testing. Exact equality of four population means is rarely a hypothesis anyone believes. If you know which comparisons matter, planned contrasts fixed in advance beat an F test followed by a post-hoc trawl.

You are testing four pricing-page layouts, and the outcome is seconds on the page before clicking through. Within-arm variability is about 45 seconds. You would act on a result where one layout runs 20 seconds from the other three — a smaller pattern than it sounds, since with three means together and one apart the SD of the four means is only about 8.7 seconds.

That 296 buys the omnibus answer and nothing else: four times in five the F test will tell you the four layouts are not all the same, and it will not identify the winner. Sizing for the comparison you actually want — layout D against control, at a Bonferroni-adjusted level across all six pairwise contrasts — takes closer to 120 per arm, because the correction pushes the per-comparison α to about 0.008 and the critical value up with it. That gap is the real cost of a four-arm test, and the moment to notice it is while planning, not when someone asks which one won.

Start with ω² and the group means, not with F. The F statistic answers a yes/no question about exact equality; ω² says how much of the outcome's variation goes with which arm a unit landed in. An ω² of 0.01 alongside p < 0.001 is ordinary in a large test: the arms differ detectably while explaining almost nothing. Prefer ω² to η², which is biased upward by roughly (k − 1)/(N − 1) even when nothing is going on.

A significant F licenses exactly one sentence: the group means are not all equal. It does not rank the arms, and the arm with the highest observed mean is not thereby the winner — with four arms and no real differences one still has to come first, and it will look convincing. A claim about a specific pair needs a procedure that controls error across the comparisons you make: Tukey's HSD for all pairwise contrasts, Dunnett's when every arm is compared against one control, Bonferroni on a short list fixed in advance. Our guide to choosing between corrections covers the trade-off; every one of them costs power.

A non-significant F is not permission to call the arms equivalent, and it is a weak statement — the omnibus test spreads its power across every way the means could differ, so it can miss a real gap between two specific arms while detecting nothing overall. If the question is whether a cheaper variant is no worse than the incumbent, nominate a margin in advance and use an equivalence test or a non-inferiority test on that pair.

Finally, treat multi-arm tests as a design problem before they become an analysis problem. Splitting fixed traffic four ways costs power twice over, and the marginal arm is usually the one nobody had a hypothesis for. Where the arms are geographies rather than users, geo testing gives the same comparison with far fewer independent units; where you just want more decisions per quarter, parallel testing beats more arms.