What is an experimentation platform?

An experimentation platform has four jobs. It assigns units to variants consistently and independently across concurrent tests. It records who was assigned and who was actually exposed. It computes results with correct standard errors for the metrics in use. And it enforces the checks that catch broken experiments before anyone reads them. The first two are engineering; the last two are where most platforms are weaker than their users assume.

The distinction from a feature-flag service is worth being clear about, because the two are often conflated and one is a subset of the other. A flag service decides who sees what. A platform adds random assignment, exposure logging, metric computation, statistical inference and validation. Teams that adopt flagging and then compute results in a notebook have the delivery half and none of the analytical half, which is where the subtle errors live.

The analytical requirements are more demanding than they look. Standard errors have to be computed at the randomization unit, which for session or event metrics means aggregating first. Ratio metrics need the delta method or a bootstrap rather than the naive formula. Both of those are commonly wrong in home-built implementations and in some commercial ones, and the failure is silent — the platform produces plausible numbers that are simply overconfident.

The guardrails are what most distinguish a mature platform, and they are cheap relative to their value. A sample ratio mismatch check on every experiment at a strict threshold. Automatic guardrail metrics that flag degradation regardless of what the experiment was about. A requirement to declare the primary metric and duration before launch, so the decision rule exists before the data. And periodic A/A tests validating the machinery itself. None is technically hard; each closes a route to a confident wrong answer.

Build-versus-buy turns on volume and integration rather than on capability. Below roughly ten concurrent experiments a commercial tool is almost always cheaper than the engineering it replaces. Above that, and particularly where assignment must happen inside backend services or where metrics live in a warehouse the vendor cannot see, building becomes reasonable — and the part worth building carefully is the analysis and guardrail layer, not the assignment, which is a hash function.

A company with 40 concurrent experiments audits its home-built platform against four requirements before deciding whether to invest further or migrate to a vendor. Each is checked against 500 simulated A/A runs on historical data.

The per-user column passing is what makes this dangerous rather than obviously broken. Anyone spot-checking the platform on a conversion metric would find it behaving correctly, and the 17.4% false-positive rate on ratio metrics would never surface — those experiments simply report significance more often than they should, which looks like a productive quarter. The assignment failure compounds it: with the experiment id missing from the hash, all 40 experiments share two populations rather than being independent, so interactions between them are systematic. The remediation order follows from cost rather than severity: adding the experiment id to the hash is a one-line change that must be done between experiments; automating the SRM check is a day's work; fixing ratio variance is a genuine project. Migrating to a vendor would solve the analytical half and not the assignment half, since that lives in their own services.