A chi-square test compares the counts observed in a table against the counts expected if the rows and columns were unrelated. The statistic adds up the squared gaps between them, so a large value means the pattern is further from independence than chance would explain.
χ²A chi-square test works on counts arranged in a table — two variants against converted and not converted, or three plans against four support-ticket categories. It computes what each cell would contain if the two variables were completely unrelated, compares that against what actually landed there, and adds up the squared discrepancies scaled by how big each expected count was. A large total means the observed pattern is hard to reconcile with independence.
The expected counts are the part worth understanding, because they are where the null hypothesis lives. If 4.5% of all users converted overall, and one arm received 30% of the traffic, then independence predicts that arm holds 30% of the conversions. The test never asks whether that prediction is sensible; it only asks how far the data strayed from it. Everything the test knows about "no relationship" is encoded in that one multiplication of row total by column total.
For a 2 × 2 table this is the same test as a two-proportion z-test, exactly: χ² equals z², and the p-values match to the last decimal. The reason to reach for chi-square anyway is that it generalises. Four variants against converted-or-not is a 4 × 2 table that a z-test cannot handle, and the chi-square statistic answers "is anything going on across these four" in one number. What it will not tell you is *which* variant differs, and chasing that down means pairwise comparisons and a multiple comparisons correction.
Its most valuable everyday use in experimentation is not measuring an effect at all — it is the sample ratio mismatch check. Assign 50/50 and observe 49.2/50.8 across 200,000 users, and a chi-square goodness-of-fit test against the intended split says whether that gap is ordinary randomness or evidence that something upstream is dropping users non-randomly. It is a cheap test that invalidates an entire experiment when it fires, and it is skipped far more often than it is run.
The condition to respect is on expected counts rather than on how many users you have: every cell should expect at least five, and a table with 100,000 rows can still fail that if one category is rare. Below the threshold the chi-square approximation to the true distribution breaks down and the p-value is not trustworthy — Fisher's exact test computes the answer directly and is the right fallback.
One sum over every cell, and one rule for the expected counts it is compared against. Degrees of freedom come from the shape of the table rather than from the number of observations.
χ² = Σ ( O − E )² / EO is the observed count in a cell, E the expected one. Dividing by E is what stops large cells from dominating purely because they are large.
E_ij = ( row total_i × column total_j ) / grand totalThe entire null hypothesis, in one line. Everything the test means by 'no relationship' is this prediction.
df = ( rows − 1 ) · ( columns − 1 )A 2 × 2 table gives 1, a 4 × 2 gives 3 — see degrees of freedom.
χ²(1 df) = z²For a 2 × 2 table these are one test in two notations. 2.63² = 6.92, and both give p = 0.0085.
every expected cell count ≥ 5About expectation, not sample size. Below it, use Fisher's exact test.
An experiment is assigned 50/50 across four weeks. The platform reports 198,340 users in control and 201,660 in the variant — a 49.6/50.4 split. It looks close enough to ignore, and the team wants to know whether it is. This is a goodness-of-fit test against the intended allocation, not a test of the metric.
p = 0.00000015. A split this uneven would occur by chance roughly once in 6.7 million identically run experiments.
This is the check earning its keep. A 49.6/50.4 split looks like nothing — well under one percent — and at this traffic it is overwhelming evidence that assignment is not doing what it claims. The cause is usually mundane and always serious: a redirect that fails more often on one variant, a bot filter applied after assignment, an event that fires late on a slower page. Whatever it is, users are being dropped non-randomly, the two arms are no longer comparable, and every metric computed from this experiment is suspect regardless of how significant it looks. The right response is to stop reading the results and find the leak. Note how the sample size drives this: the same 0.83% deviation on 4,000 users gives χ² = 0.28 and p = 0.60, which is genuinely unremarkable.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.