Standard error measures how much an estimate would vary if you repeated the study. It describes the precision of a statistic rather than the spread of the data, and it shrinks as the sample grows — which standard deviation does not.
SERun your experiment again on a fresh sample and you would get a slightly different number. Run it a thousand times and those numbers would form a distribution around the truth. The standard error is the standard deviation of that distribution — how much your estimate bounces around from sample to sample. It is a statement about the precision of a statistic, not about the variability of the underlying data, and that distinction is the one people most often lose.
Standard deviation describes the spread of individual observations and does not change as you collect more of them: customers vary as much in a sample of a million as in a sample of a hundred. The standard error describes how well you have pinned down a summary of those observations, and it falls as 1/√n. Confusing the two produces confidence intervals wrong by a factor of the square root of the sample size — a mistake that is invisible in the output and enormous in effect.
That square root is the source of most of the economics of experimentation. Four times the data halves the standard error; a hundred times the data reduces it by only a factor of ten. This is why detecting small effects is so expensive, why sample size scales with the inverse square of the effect you want to detect, and why reducing the variance of the metric is so much more attractive than collecting more of it — the numerator and the denominator are not equally easy to move.
It is also the engine of everything you read off a results table. A confidence interval is the estimate plus and minus a critical value times the standard error. A test statistic is the estimate divided by its standard error. So every significance verdict and every interval is really a statement about how many standard errors the estimate sits from zero — which means anything that makes the standard error wrong makes both of them wrong in the same direction.
Two situations routinely do exactly that. Clustered data: if observations within a group are correlated — sessions from the same user, weeks from the same market — the effective sample size is far smaller than the row count, and treating rows as independent understates the standard error badly. And repeated looks at accumulating data, where the arithmetic is fine but the decision rule inflates the error rate. The first is fixed by clustering at the level of assignment; the second by sequential testing.
One general form and the three specialisations you will actually use. Every one of them has n under a square root.
SE = s / √ns is the sample standard deviation. The √n is why quadrupling the sample only halves the uncertainty.
SE = √( p̂(1 − p̂) / n )Largest at p = 0.5 and smallest at the extremes — though very rare events need the Wilson interval instead, since this approximation fails there.
SE = √( s₁²/n₁ + s₂²/n₂ )Variances add, standard errors do not. This is the quantity every two-sample test divides by — see the two-sample t-test calculator.
SE_clustered ≈ SE_naive · √( 1 + (m − 1)·ρ )m is the cluster size and ρ the within-cluster correlation. With 10 sessions per user at ρ = 0.3, the naive standard error is about 1.8 times too small.
An experiment measures revenue per session. There are 200,000 sessions per arm, from 40,000 users — an average of five sessions each. Sessions from the same user are correlated at about 0.35. The analysis was run at session level.
The correct standard error is £0.0318 × √2.4 = £0.0492 — about 55% larger. The confidence interval is 55% wider and the test statistic 35% smaller.
Nothing about the data is wrong; the analysis simply counted 200,000 observations where it had roughly 83,000 observations' worth of independent information. A result at p = 0.03 under the naive standard error lands at about p = 0.09 once clustering is accounted for, which is the difference between shipping and not. The rule that avoids this is to analyse at the unit you randomised on: users were assigned, so the analysis should be per user, either by aggregating each user's revenue into one number or by clustering standard errors at the user. Randomising by user and analysing by session is one of the most common ways an experimentation platform quietly reports more significance than it has earned.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.