SUTVA is the assumption that a unit's outcome depends only on its own treatment, and that the treatment means the same thing for everyone. Every standard A/B test relies on it, and it fails quietly wherever users can affect each other.
SUTVA — the stable unit treatment value assumption — is the condition that makes a two-group comparison mean what it appears to mean. It has two halves. First, no interference: one unit's treatment assignment does not affect another unit's outcome. Second, no hidden variations of treatment: everyone assigned to a variant receives the same version of it. Both are assumed by every standard experiment and neither is usually checked.
The first half is the one that fails most often and most consequentially. If treated users can affect untreated ones — through a shared marketplace, a collaborative document, a social feed, or simply by telling a colleague — then the control group is partly treated and the comparison understates the effect. That failure is called interference, and its distinguishing property is that it attenuates towards zero, so it hides real effects rather than manufacturing false ones.
The second half fails more subtly. "The treatment" is supposed to be one thing, and often is not: a feature that renders differently on old browsers, an email delivered at different times across time zones, a recommendation model that behaves differently for users with sparse histories. The estimated effect is then an average over versions rather than the effect of a single intervention, which is not wrong so much as ambiguous — you cannot say what would happen if you shipped it, because it is several things.
Neither half is testable from the data. There is no diagnostic that reports a SUTVA violation, because the violation concerns what would have happened under a different assignment — the counterfactual again. What you can do is reason about the mechanism: ask whether a route exists by which a control user could be affected by the treatment, and whether the treatment is genuinely uniform. That is a product question rather than a statistical one, which is why it is so often skipped.
The remedy when it fails is to change the randomisation unit rather than the analysis. Cluster randomization assigns whole groups so spillover stays inside an arm; a switchback test randomises time when the whole system is shared. Both cost substantial statistical power, which is the honest price of an unbiased estimate — and a wide interval around the right quantity is worth more than a narrow one around the wrong one.
The assumption stated formally, and what its two halves rule out.
Yᵢ( D ) = Yᵢ( Dᵢ )Unit i's outcome depends on the whole assignment vector D only through its own element Dᵢ.
Yᵢ does not depend on Dⱼ for any j ≠ iFails in marketplaces, social products and shared workspaces — see interference.
Dᵢ = 1 means the same treatment for every iFails when rendering, timing or model behaviour differs across users. Makes the estimand ambiguous rather than biased.
observed effect ≈ true effect × ( 1 − contamination )Attenuates towards zero, so real effects are hidden — see the sample size calculator.
A collaboration tool tests a new commenting feature with user-level randomisation across 40,000 users in 3,100 workspaces. Because comments are visible to everyone in a document, control users in mixed workspaces encounter the feature through their colleagues. The team re-runs it clustered by workspace.
The user-level design measured 1.9%. The clustered design measured 5.1% — the user-level estimate was attenuated to roughly a third of the true effect.
The middle rows show the mechanism directly. Seven in ten control users shared a workspace with someone who had the feature, so they saw the comments and were partly treated; comparing against the 29% who were genuinely untreated recovers something close to the clustered answer. That comparison is a useful diagnostic and not a valid analysis, because pure-control workspaces are not a random subset — they are the smaller ones. The clustered rerun is the defensible estimate, and note the cost: its interval spans 1.2% to 9.0%, far wider than the user-level design produced, because 3,100 workspaces carry much less information than 40,000 users. That is the trade SUTVA violations force. The user-level test was precise about the wrong quantity, and precision about a contaminated comparison is worth nothing.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
Our “Geo Testing: Unlocking True Incrementality” webinar explored how teams can measure real-world impact when A/B testing isn’t possible - using geographic experiments and synthetic controls to reveal true incremental lift in marketing and product initiatives.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.