What is interference?

An A/B test compares two groups on the assumption that each user's outcome depends only on their own assignment. Interference is the violation of that assumption — treating one user changes another user's outcome — and it is the practical form in which SUTVA fails. Spillover is the same thing under a different name, and the two words are used interchangeably.

The routes are usually obvious once looked for. In a marketplace, treated buyers consume supply that untreated buyers would have had, so a change that wins by capturing scarce inventory looks good and creates nothing. In collaboration software, a treated user's output is visible to untreated colleagues. In social products, activity propagates through the feed. And in advertising, a household sees the same television whoever was assigned.

The direction of the bias is the useful thing to know: contamination of the control group pulls the two arms together, so interference attenuates the measured effect towards zero. It hides real effects rather than manufacturing false ones. That makes it dangerous in a specific way — a team runs a test, sees nothing, concludes the idea does not work, and the experiment was simply incapable of seeing it. The failure looks like a clean negative result.

There is one important exception where the direction reverses. In a marketplace with fixed supply, treatment can win by taking from control rather than by creating anything, so the measured difference overstates what would happen if the change shipped to everyone — because at full rollout there is nobody left to take from. Both failure modes come from the same broken assumption, and which one you get depends on whether the interaction is sharing or competing.

The remedy is always to enlarge the randomisation unit until the spillover is contained inside it. Cluster randomization by workspace, household or market handles social and collaborative leakage; a switchback test handles a fully shared system by randomising time instead of units. Both cost substantial power, and that cost is what buys an estimate of the right quantity rather than a precise estimate of the wrong one.

A delivery marketplace tests a courier-matching change with user-level randomisation. Treated orders are matched more aggressively, drawing from the same courier pool as control orders. The team compares the user-randomised result against a switchback design on the same weeks.

This is the competing case rather than the sharing one, so the bias runs the other way and the naive design overstates rather than attenuates. Treated orders were matched to couriers that control orders would otherwise have received, so part of the measured gain is redistribution: control got worse because treatment got better. At full rollout every order is treated and there is nobody left to take from, which is why the switchback figure of 1.6 minutes is the one that predicts what shipping does. Note how convincing the wrong answer looks — a 17% improvement with a tight interval on 40,000 orders. Nothing in the user-randomised output hints at a problem, and the only way to know is to reason about whether the arms compete for a shared resource. Where they do, the design has to change before the numbers can be believed.