Paste a table of counts and get the Pearson chi-square statistic, its degrees of freedom, the p-value, every expected count and Cramér's V. It handles both jobs the test does: independence between two categorical variables, and goodness-of-fit of one variable against a set of expected proportions. The sample-size tab sizes the table in advance from Cohen's w.
| Observations in total | 964 |
|---|---|
| Degrees of freedom | 2 |
| Achieved power | 80.0% |
| Critical value | 5.991 |
| Non-centrality (λ) | 9.640 |
Collect 964 observations in total — spread across the whole table, not 964 per row — and the test will detect an association as strong as Cohen's w of 0.100 about 80.0% of the time, on 2 degrees of freedom. A larger table needs more for the same effect, because it gives chance more ways to look uneven. The requirement scales as 1/w², so halving the association you want to catch quadruples the data — and what you get for it is still only the omnibus answer, never which category was responsible.
| Power | Sample size |
|---|---|
| 5.1% | 1 |
| 18.9% | 161 |
| 34.2% | 321 |
| 48.7% | 481 |
| 61.4% | 641 |
| 71.8% | 801 |
| 79.9% | 961 |
| 86.0% | 1,121 |
| 90.4% | 1,281 |
| 93.5% | 1,441 |
| 95.7% | 1,601 |
| 97.2% | 1,761 |
| 98.2% | 1,921 |
| Effect size (Cohen's w) | Required observations in total |
|---|---|
| 0.05 | 3,854 |
| 0.11 | 851 |
| 0.16 | 364 |
| 0.22 | 201 |
| 0.28 | 127 |
| 0.33 | 88 |
| 0.39 | 64 |
| 0.44 | 49 |
| 0.50 | 39 |
| 0.56 | 31 |
| Column 1 | Column 2 | |
|---|---|---|
| Row 1 | ||
| Row 2 | ||
| Row 3 |
| p-value | 0.0132 |
|---|---|
| χ² | 8.6613 |
| Degrees of freedom | 2 |
| Cramér's V | 0.0380 |
| Pearson χ² (uncorrected) | 8.6613 |
| Observations | 6,000 |
| Smallest expected count | 215.00 |
The table is not independent (p 0.0132 on 2 degrees of freedom) — and that is the entire claim. The test does not say which cell drove it, and reading the largest-looking gap off the grid afterwards is a search, not a result; a specific pair needs its own comparison with the multiplicity accounted for. Cramér's V is 0.038, which is close to nothing: which category a unit fell into explains a negligible share of the outcome, and it is the number to report next to the p-value: χ² grows with the sample while the pattern stays put, so a big enough table makes a trivial association significant. V is computed from the uncorrected Pearson statistic, and stays that way if you turn Yates' correction on.
| Cell (row × column) | Observed | Expected |
|---|---|---|
| R1C1 | 210 | 215 |
| R1C2 | 1790 | 1785 |
| R2C1 | 246 | 215 |
| R2C2 | 1754 | 1785 |
| R3C1 | 189 | 215 |
| R3C2 | 1811 | 1785 |
Reach for chi-square when both variables are categorical and you want to know whether they are related at all: channel by plan tier, subject line by opened-or-not, device by drop-off step, region by churn reason. The test compares what you observed against what independence would have produced given the row and column totals, and asks whether the gap is bigger than chance. Goodness-of-fit is the same machinery with one row — traffic across five landing pages against the split you expected.
The result is deliberately blunt. A significant chi-square says "something in this table is not independent"; it does not say which cell, which pair, or in which direction. On a 2×2 table that is unambiguous, and the test is identical to the two-proportion z-test — squaring the z gives the chi-square exactly. Anything larger needs a follow-up, and that is where discipline is required: comparing every pair of rows after an omnibus test is precisely what multiple comparison corrections exist for.
Some tables want a different test. When counts are small — any expected count below about five — the approximation breaks and Fisher's exact test computes the answer instead. When the same subjects appear in both rows and columns, as in a before-and-after survey, the data is paired and McNemar's test is correct; chi-square treats it as independent and understates the evidence. When the outcome is a number rather than a category, one-way ANOVA compares means. And when it is "how long until it happened", the table has discarded the timing and censoring that the log-rank test preserves.
Three of these are ritual and one is real. The expected-count rule gets the textbook attention; independence is what actually invalidates results.
The statistic is Pearson's: for each cell, the squared gap between observed and expected, scaled by expected, summed across the table. Under independence it follows a chi-square distribution on (r−1)(c−1) degrees of freedom, and the p-value is the upper tail. Because every deviation is squared the test is inherently two-sided — there is no one-tailed chi-square. Sample size runs the same distribution non-centrally: you give an effect size as Cohen's w, and the calculator finds the total N at which the non-central chi-square clears the critical value with your chosen probability. Cramér's V sits next to the p-value on purpose, because on a large table the p-value alone makes a negligible association look like a finding — the failure mode described in [reporting results properly](/post/statistically-speaking-how-to-properly-report-a-b-testing-results).
Eᵢⱼ = (row totalᵢ × column totalⱼ) / NWhat the cell would hold on average if the two variables were unrelated and the margins were as observed. For goodness-of-fit it is simply N × pᵢ, with pᵢ the hypothesised proportion.
χ² = Σᵢⱼ (Oᵢⱼ − Eᵢⱼ)² / EᵢⱼDividing by the expected count is what makes a gap of 30 in a cell expecting 100 count for far less than the same gap in a cell expecting 10.
df = (r − 1)(c − 1) [independence]; df = k − 1 [goodness-of-fit]Independent of N. Adding observations makes a given χ² more significant; adding categories makes it less, because a bigger table has more ways to look uneven by chance.
V = √( χ² / (N · min(r − 1, c − 1)) )The effect size: 0 to 1, and free of sample size. On a 2×2 table it equals the phi coefficient, the correlation between the two binary variables. Interpret it in context, but treat anything under about 0.1 as too weak to build a decision on however small the p-value.
w = √( Σ (p₁ᵢ − p₀ᵢ)² / p₀ᵢ ) and w = V·√(min(r−1, c−1))Cohen's benchmarks are 0.1 small, 0.3 medium, 0.5 large — benchmarks, not standards. A w of 0.05 can be commercially enormous on a table of a million sessions.
N = λ(df, α, 1−β) / w²λ is the non-centrality at which a non-central chi-square on df degrees of freedom clears the critical value with the target power. At α = 0.05 and 80% power it is about 7.85 for df = 1, 9.63 for df = 2 and 10.90 for df = 3 — so a 3×2 table needs roughly 964 observations to detect w = 0.1.
You tested three subject lines on a newsletter, 2,000 sends each, and want to know whether click-through differs before anyone declares a winner. A got 210 clicks, B got 246 and C got 189 — 10.5%, 12.3% and 9.45% against an overall 10.75%.
χ² = 8.66 on 2 degrees of freedom, p = 0.013. Expected clicks are 215 in every row, so no cell is anywhere near the small-count warning. Cramér's V = 0.038.
The subject lines are not interchangeable — a gap this size across three arms of 2,000 turns up by chance about once in seventy-five. But V = 0.038 says the association is close to nothing: which line someone received explains a negligible share of whether they clicked. Both are true at once, and reporting only the first is how a three-point relative difference becomes a breakthrough in a deck. The test also names no winner: the omnibus p-value covers the whole table, and showing that B genuinely beats C needs a pairwise comparison with the multiplicity accounted for — a comparison a good deal less impressive than 0.013 sounds.
Read the effect size and the p-value together, and let the effect size lead. Chi-square is unusually good at producing significance from nothing: the statistic grows linearly with N while the pattern in the table stays put, so any table with a real but trivial association crosses 0.05 once the sample is large enough. Cramér's V is invariant to that. When V is 0.03 and p is 0.001, the honest summary is "a real association, too small to act on".
Then look at the cells. An omnibus p-value says the table is not independent; it does not say where. Standardised residuals — each cell's signed contribution to χ² — show which cells drive the result, and a table where one row supplies nearly all the statistic tells a different story from one where every cell is slightly off. Use them to form the follow-up question, not as p-values in their own right.
Be honest about the direction of inference. Chi-square detects association, and association in observational counts is almost always partly confounded. A table showing that paid-social users churn more is equally compatible with paid social attracting different people; separating the two needs a design built for it, or a randomised comparison sized with the A/B test calculator.
Finally, mind the small-count warning rather than working around it. If several expected counts sit below five the p-value is approximate in an unhelpful direction, and the usual instinct — merging categories until the warning disappears — changes the hypothesis being tested. For a 2×2 table Fisher's exact test removes the problem; for larger sparse tables, collapse on grounds decided in advance, or use a simulated p-value.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
We turn tables of counts into decisions, with the effect size reported next to the p-value rather than instead of it. A/B Testing