A paired t-test compares two measurements taken on the same units — the same users before and after, or the same markets under two conditions. It analyses the differences within each pair, which removes variation between units from the comparison entirely.
Some comparisons come with a natural partner for every observation. The same users measured before and after a change; the same geographic markets under two media plans; the same accounts scored by two models. In all of those the two numbers being compared share whatever makes that unit distinctive, and a paired t-test exploits that instead of throwing it away.
The mechanism is almost embarrassingly simple: compute the difference within each pair, then run a one-sample t-test on those differences against zero. What that accomplishes is not simple at all. Between-unit variation — the fact that some users are heavy and some are light, some markets large and some small — vanishes from the calculation, because it is present in both halves of every pair and subtracts out. What remains is only the variation in how much each unit *changed*, which is usually far smaller.
That reduction is the entire value, and it can be dramatic. If users vary enormously in baseline spend but the treatment moves everyone by roughly the same amount, the unpaired comparison is drowning in between-user noise while the paired one sees a clean signal. Sample-size savings of five to ten times are ordinary in that situation. The relevant quantity is the correlation between the two measurements: the higher it is, the more a pairing buys, which is exactly the logic CUPED applies to a pre-experiment covariate rather than a second measurement.
The reason this is not the default in online experimentation is that pairing usually is not available. A standard A/B test shows each user one variant, so there is no second measurement to pair with — the arms contain different people and the comparison is unavoidably between units. Attempting to manufacture pairs by matching users on observed characteristics is not the same thing and reintroduces every problem propensity score matching has. Where pairing genuinely does appear is geo testing, where markets are matched on pre-period behaviour, and in before-and-after designs, which carry their own threats from seasonality and trend.
There is one honest cost. A paired design halves the degrees of freedom compared with an unpaired test on the same total number of measurements, since n pairs give n − 1 rather than 2n − 2. When the pairing is weak — the two measurements barely correlate — that loss is not repaid and the paired test is genuinely worse. Pairing on something irrelevant is not a free option.
A one-sample test on the within-pair differences. Everything that makes it powerful is contained in the fact that the differences, not the raw values, are what get analysed.
dᵢ = x₁ᵢ − x₂ᵢ, for each pair iThis single step is what removes between-unit variation. Everything after it is a one-sample test.
t = d̄ / ( s_d / √n )d̄ is the mean difference, s_d its standard deviation, n the number of PAIRS — see the paired t-test calculator.
df = n − 1Pairs minus one, not observations minus two. Half what an unpaired test on the same measurements would have.
Var(d) = σ₁² + σ₂² − 2ρσ₁σ₂The whole case for pairing. At ρ = 0.9 with equal variances this is a fifth of the unpaired variance; at ρ = 0 it is larger.
A geo test runs a campaign in 24 matched market pairs, one treated and one held back in each pair. Weekly revenue per market averages £182,000 with a standard deviation of £96,000 — markets differ enormously in size. The treatment lifts revenue by £4,100 per market on average, and the pre-period correlation within pairs is 0.94.
Paired: t(23) = 2.42, p = 0.024. Unpaired: p = 0.88, no effect detectable at all.
The same data, analysed two ways, and only one of them can see anything. Unpaired, the £4,100 effect is buried under a £96,000 standard deviation between markets — London and Hull differ far more from each other than either differs from itself under two media plans. Pairing removes that entirely and leaves an £8,300 spread in how much each pair moved, against which £4,100 is visible. This is why geo tests are built on matched markets rather than randomly split ones, and it is also where the design's fragility lives: the whole gain depends on the 0.94 correlation holding into the test period. If a matched pair diverges for an unrelated reason — one market has a store closure, a competitor enters the other — the pairing that bought this precision is also what carries the contamination straight into the estimate.
In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.
Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.