What is a feature flag?

A feature flag separates deploying code from releasing behaviour. The new path ships to production disabled, and a runtime configuration decides who sees it — everyone, nobody, 10% of users, or one arm of an experiment. That separation is what makes continuous deployment compatible with careful release, and it is the delivery layer nearly every experimentation platform is built on.

Three distinct jobs run on the same infrastructure and they are worth keeping separate in your head. A release flag rolls something out gradually, watching error rates as the percentage climbs — that is risk management, not measurement, and it has no control group. An experiment flag assigns users randomly and holds the assignment stable for the duration, which is what makes a comparison valid. A kill switch exists to turn something off immediately when it misbehaves. Conflating the first two is the common error: a gradual rollout tells you the system did not fall over, and nothing about whether the change was any good.

For experimentation the requirements on the flag are specific. Assignment must be deterministic — the same user gets the same variant on every request and every device, or their experience is inconsistent and their data uninterpretable. It must be based on a stable identifier rather than a session, and it must be recorded, since an experiment whose assignments were not logged cannot be analysed at all.

The mistake that does the most damage is changing a flag's configuration mid-experiment. Increasing the rollout from 10% to 50% halfway through means the later users were assigned under different conditions and joined at a different point in the calendar, so the two arms are no longer comparable — this is a common cause of sample ratio mismatch, and it usually invalidates everything collected before the change. If the allocation must change, the honest response is to restart the experiment.

There is also an operational cost that experimentation teams inherit. Flags accumulate: every one is a branch in the code, and a codebase with hundreds of stale flags has an untestable number of possible states. Flags that outlive their purpose should be removed and the losing branch deleted, which sounds obvious and is one of the most consistently neglected pieces of hygiene in experimentation-heavy organisations.

A team launches an experiment at a 10% allocation to watch for errors, sees none after two days, and raises it to 50% to finish faster. The test runs a further twelve days and reports a 4.1% lift on conversion. A reviewer runs a sample ratio check before it ships.

The imbalance comes from the reallocation rather than from any bug in the assignment layer. When the percentages changed, the hashing boundaries moved, so some users who had been in control on day two found themselves in treatment on day three — and those users carry two days of control behaviour into the treatment arm's data. The arms now differ in composition and in when their members joined, and there is no analysis that separates that from the treatment effect. Two things to take from it. The instinct behind the change was sound: starting small to watch for errors is good practice. The error was doing it inside a running experiment rather than as a separate release phase that concludes before the experiment starts. And the SRM check is what caught it — a 3.4% imbalance is invisible by eye and overwhelming statistically, which is exactly why that check should run automatically on every experiment rather than when someone thinks to look.