What is minimum detectable effect (MDE)?

Before you can compute a sample size, you have to answer a question statistics cannot answer for you: how small an improvement would still be worth having? That number is the minimum detectable effect. Set it at a 5% relative lift and the experiment is built to find a 5% lift four times in five; a 2% lift, if one exists, will mostly slip past unnoticed. The MDE is therefore not a prediction of what will happen — it is a statement about the resolution of the instrument you are building.

It should come from value, not from history. The right question is what improvement would justify the cost of designing, building, shipping and maintaining the change, and any lift below that is one you would not act on even if you measured it perfectly. Working the other way round — looking at past experiments and picking a plausible-sounding number — produces tests powered for whatever tends to happen rather than for what you need to know, and it is how teams end up unable to act on their own results.

The square law makes this choice expensive to get wrong in one direction. Because sample size scales with the inverse square of the effect, halving the MDE quadruples the traffic required. So a team that sets a needlessly ambitious MDE of 1% will find that no test is ever feasible, and a team that inflates it to 15% to make the numbers work will run tests that cannot detect anything they would actually ship. The honest version of the second case is to say out loud: this test can only find a 15% lift, and a flat result rules out nothing smaller.

That last sentence is the MDE's most useful function, and it is usually left unwritten. A non-significant result does not mean no effect — it means no effect *of at least the detectable size* was found. Reporting the MDE alongside a null result converts an unhelpful "it did nothing" into a precise "we can rule out anything above 8%", which is a real finding. The confidence interval does this job more directly on the actual data, and the two should be read together.

One nuance that trips people up: the MDE is not a threshold the observed effect must clear. A test designed for a 5% MDE can perfectly well detect a 12% effect — it will do so almost every time — and it can also, with luck, return significance on a 3% one. The MDE describes the effect size at which power equals the level you chose, usually 80%. Above it, power is higher; below it, lower; nothing is impossible on either side.

A team has 30,000 users per arm available over a two-week window on a metric with a 6.0% baseline. Rather than asking how long a test would take, they ask the more useful question: what is the smallest effect this window can actually detect, at 5% two-sided and 80% power?

This test should not be run as designed, and the calculation takes five minutes rather than two weeks to establish that. A 9% detectable effect against a 3% expected one means the most likely outcome by far is a flat result that says nothing — the experiment cannot distinguish a change worth shipping from no change at all. Three responses are honest. Extend to eight weeks, which quadruples the sample and brings the MDE to about 4.5%. Apply variance reduction and re-run the calculation. Or reframe the test as a directional read and agree in advance that it cannot decide anything, which is occasionally reasonable and should always be written down. Running it and reporting a null as evidence of no effect is the one option that is not available.