What is propensity score matching?

You want to know whether joining the loyalty programme increases spend. You cannot randomise it, because joining is voluntary, and joiners already differ from non-joiners in every way that matters. Propensity score matching attempts a repair: build a model predicting who joins from their observable characteristics, then compare each joiner with a non-joiner who had almost the same predicted probability of joining. Within those pairs, membership is at least not predictable from the variables you measured.

The idea that makes it practical is a dimension reduction. Matching directly on twenty covariates is hopeless — nobody has an exact twin on all of them — but Rosenbaum and Rubin showed that matching on a single number, the probability of treatment given those covariates, achieves the same balance. So the workflow is: fit a model for treatment, obtain each unit's propensity score, match on that score, and estimate the effect within the matched sample.

The critical assumption is that treatment is as good as random once you condition on the measured variables. In a loyalty example, this says two customers with the same age, tenure, prior spend and channel mix are equally likely to join for reasons unrelated to their future spending — which is almost certainly false, because the thing driving both joining and spending is enthusiasm, and enthusiasm is not in the table. There is no test for this. A confounding variable you never measured is invisible to every diagnostic the method offers.

What you must check is balance, and it is a genuine check rather than a formality. After matching, every covariate should be distributed similarly across the two groups; the standard summary is the standardised mean difference, and the conventional threshold is 0.1. If balance fails, the specification is wrong and the estimate is not usable — a routine outcome, and the reason a matching analysis should always report a before-and-after balance table. Also check overlap: treated units whose propensity scores fall outside the range of any untreated unit have no counterfactual and should be trimmed, which narrows the population your estimate describes.

Modern practice has largely moved past one-to-one matching. Inverse probability weighting uses the same scores to reweight the whole sample rather than discarding unmatched units. Doubly robust estimators combine a propensity model with an outcome model and remain consistent if either is correct, which is a genuine improvement. And matching on covariates directly, or on a coarsened version of them, avoids known pathologies of propensity matching. All of them share the same untestable assumption, which is why the honest framing is that this method buys its causal claim with an argument, while randomization buys it with design.

An analyst evaluates a loyalty programme. Members spend £412 a year against £268 for non-members. They fit a propensity model on age, tenure, acquisition channel, prior-year spend and category mix, match one-to-one on the score, and compare.

The balance table is the part worth dwelling on. Prior-year spend differed by 0.74 standard deviations before matching, which means the raw comparison was mostly comparing high spenders with low spenders and calling the difference a programme effect. After matching it is 0.06, so the observable confounding has been dealt with properly. What remains is the question no diagnostic can answer, and the sensitivity figure quantifies it: an unmeasured variable that raised the odds of joining by only 30% would be enough to explain away the £47 entirely. Enthusiasm plausibly does much more than that. The defensible way to report this is £47 with the sensitivity attached — and the way to actually settle it is to randomise the programme invitation.