What is a z-test?

A z-test has the same shape as every other significance test: take the difference you observed, divide it by how much that difference would bounce around under chance alone, and see how far out on a reference curve the result lands. What makes it a z-test rather than a t-test is that the reference curve is the standard normal — the ordinary bell curve — rather than the slightly heavier-tailed t.

The textbook reason for choosing between them is that a z-test applies when the population standard deviation is known and a t-test when it has been estimated from the sample. In practice that distinction decides almost nothing, because the population standard deviation is essentially never known. The reason z-tests are everywhere in experimentation is different and much more specific: for a proportion, the variance is not a separate unknown at all. A conversion rate of p has variance p(1 − p) by construction, so there is nothing left to estimate and nothing to inflate the tails for.

That is why the standard readout on an A/B test of conversion is a two-proportion z-test. Control converted 2,150 of 50,000, variant converted 2,310 of 50,000, and the difference between those two rates is compared against a standard error built from the pooled rate. It is the same arithmetic a chi-square test performs on a two-by-two table — for that specific case the two are algebraically identical, with z² equal to the chi-square statistic — and the choice between them is a matter of which output you would rather read.

The condition that actually matters is whether the normal approximation holds, and it is about counts rather than sample size. The usual rule is at least five expected successes and five expected failures in each arm, with ten being safer. A test with 80,000 users and a 0.004% conversion rate fails that comfortably despite the enormous sample, because it has three conversions. When the counts are that thin the approximation breaks in the direction that matters — it understates how much a rate can move by chance — and Fisher's exact test is the honest answer.

For means rather than proportions the practical advice is simply to use a t-test. At the sample sizes online experiments run at the two are numerically almost indistinguishable, since the t-distribution converges on the normal as degrees of freedom grow, and the t-test is the one that stays correct when the sample turns out smaller than planned.

A pricing page test measures signup rate. Control converted 412 of 18,400 visitors; the variant converted 486 of 18,250. That is 2.24% against 2.66%, an 18.8% relative lift, and the question is whether a gap that size is more than the split alone would produce.

A gap this large would appear by chance about once in 118 identical experiments, so the difference is real. Where it needs care is the relative framing: 18.8% sounds transformative, and the interval says the true lift is somewhere between roughly 5% and 33% in relative terms. Both ends are worth having, but a business case built on 18.8% is built on the midpoint of a wide range rather than on a measured quantity. The count condition is met with room to spare here, which is the check that would have mattered had this been a 0.05% conversion rate at the same traffic — there the same arithmetic would have run happily and produced a p-value not worth reading.