What are network effects?

A network effect exists when each additional user makes the product more valuable to everyone else. A messaging app is useless to one person and indispensable to a company where everyone is on it; a marketplace with more sellers attracts more buyers, which attracts more sellers. The property is about the product rather than about measurement — but it creates a measurement problem that no amount of statistical care resolves.

The problem is that an experiment tests a variant at partial adoption and the launch happens at full adoption. If a feature's value depends on how many of your contacts also have it, then a 50% test measures the feature at 50% penetration, and the launch delivers it at 100%. Those are different products. The experiment is not biased in the usual sense — it correctly estimates the effect at the tested scale — it just estimates a quantity that is not the one the decision needs.

The direction is almost always understatement for genuinely network-dependent features, and the size of the gap depends on how steeply value rises with adoption. A collaboration feature that requires both parties is worth roughly nothing when only half of each pair has it, so a user-randomised test can measure close to zero on something transformative. This is one of the few situations where a null experimental result is genuinely weak evidence of no effect.

It is worth separating this from interference, which the two are frequently conflated with. Network effects are the product property: value grows with adoption. Interference is what that property does to the arithmetic of your experiment by carrying treatment across the boundary between arms. One is a business phenomenon, the other is the measurement failure it causes — and keeping them distinct makes it clearer that the response is a design change rather than a different feature.

The designs that recover the real quantity all work by making the treated region large enough to contain the network. Cluster randomization by workspace or friendship group gives full adoption inside each treated cluster. Geographic randomisation does the same at market scale. Both cost heavily in power, and a staged rollout with careful before-and-after measurement is sometimes the only feasible option — with all the weaknesses that implies for the counterfactual.

A messaging product tests a feature that only works when both people in a conversation have it. It is run first as a 50/50 user-level test, then as a cluster-randomised test over 2,400 organisations.

The second and third rows explain the first entirely. At a 50/50 split only a quarter of conversations had the feature on both sides, and in half of them one party had something the other could not use — which is arguably worse than not shipping it. The user-level test correctly measured the feature at 50% penetration, and that quantity is close to zero because the feature does not work at 50% penetration. It is not a false negative in the usual sense; it is an accurate answer to a question nobody wanted asked. The clustered design gives every user in a treated organisation the feature, so treated clusters experience it as they would after launch, and the 6.8% is the number that predicts the rollout. The cost is visible in the interval — 2,400 clusters is far less information than 40,000 users — and it is worth paying because the alternative was a precise zero.