What is the novelty effect?

Move a button, change a colour, add a badge, and existing users notice. Some of them click it because it is different, not because it is better. That extra engagement is real in the data and temporary in the world, and an experiment that stops while it is still happening reports it as a lift. This is the novelty effect, and it is the single most common reason a shipped win fails to reproduce in the following quarter.

The signature is a decaying treatment effect. Plot the daily lift rather than the cumulative one and a genuine improvement holds roughly steady while a novelty effect starts high and slides towards zero. That plot is the cheapest diagnostic available and is rarely made, because the cumulative number — which averages the inflated early days into everything after — looks reassuring throughout and only drifts slowly downward.

The cleaner diagnostic is segmentation by exposure history. Novelty requires a prior expectation to violate, so it can only affect users who have seen the old version. New users have no baseline and cannot experience a change as novel. If the lift is concentrated in returning users and absent among first-time visitors, that is close to conclusive — and it also tells you what the long-run effect will be, since new users are showing you the steady state directly.

The mirror image is the hangover effect, where the disruption of change makes users temporarily worse off before they adapt, and a test stopped early reports a loss that would have recovered. Both are the same underlying phenomenon — a transient response to change rather than a durable response to the change's quality — and both are why running to a pre-registered duration matters more than reaching significance.

Two practical defences. Run long enough for the effect to decay, which in most consumer products means at least two full weeks and preferably a month for changes to a familiar surface. And where the stakes justify it, keep a long-term holdout that never receives shipped changes, so the accumulated effect of a year of wins can be measured against a genuine counterfactual rather than assumed. A programme where every test reports a lift and the aggregate metrics are flat usually has this problem somewhere in it.

A news app redesigns its article cards and tests for four weeks on articles opened per session. The team would normally have stopped at the end of week one, when the result was strongly significant. They ran the full duration and segmented by whether the user had visited before the test began.

The decay across the four weeks is the whole story, and it is invisible in the cumulative figure, which still reads +5.1% at the end and looks like a solid win. Two details make the diagnosis firm. The weekly effect flattens rather than continuing to fall, which suggests a small genuine improvement underneath the novelty rather than nothing at all. And by week four the new-user and returning-user effects have converged at around +2%, which is what convergence should look like once the transient has passed — early in the test those two numbers were far apart. The practical conclusion is that this change is roughly neutral and probably slightly positive, which is a defensible thing to ship and a very different claim from the one week one supported. Note also that the honest reading of the week-four p-value is not "no effect" but that four weeks was not enough traffic to resolve a 2% effect.