Statistics glossary

Terms from experimentation, causal inference and marketing measurement, defined properly. Each one gets its own page: what it means, the formula, a worked example, and the misreadings that cost people money.

Most statistics glossaries define a term using four other terms you also do not know. These do not. Every entry opens with a definition you can read out loud in one breath, then explains the same idea again at length, shows the arithmetic on real numbers, and finishes with the specific ways the term gets misused — because in practice the damage is almost never done by people who have never heard of a p-value. It is done by people who have heard of one and think it is the probability the result is a fluke.

The terms are grouped by the job you are doing rather than by statistical family, and each group is a page of its own. If you are running an experiment and something looks wrong, start with experimentation — sample ratio mismatch is the first thing to rule out. If you are trying to read a result somebody handed you, start with inference: p-value, confidence interval and statistical power between them explain most of what a results table is claiming.

If the question is whether your marketing actually caused anything, that is causal inference and modelling — incrementality, geo experiments and marketing mix modelling are the three tools most companies end up choosing between, and the entries say plainly what each one can and cannot tell you.

Where a term has a calculator, the entry links to it, so you can go from the definition to a number without leaving the site. We build measurement systems for a living — see our A/B testing work or the case studies — and this is the vocabulary those projects run on.

Running experiments
The vocabulary of A/B testing: what you fix before a test starts, what you watch while it runs, and the diagnostics that say the result cannot be trusted.
A/B testing, Control group, Feature flag, A/A test and 16 more
Reading a result
What a significance test actually claims, what it does not, and the four numbers — p, alpha, power, effect size — that decide whether a finding means anything.
P-value, Confidence interval, Statistical power, Minimum detectable effect and 8 more
Choosing a test
Which significance test the data in front of you calls for, what each one assumes, and what breaks when the assumption does not hold.
T-test, Z-test, Chi-square test, ANOVA and 12 more
Metrics and measurement
Choosing what an experiment is judged on: the one metric that decides, the ones that protect you, and the ones that only look like progress.
Primary metric, Guardrail metric, North star metric, Conversion rate and 9 more
Variance reduction and efficiency
Getting a usable answer from less traffic: removing predictable noise, taming the outliers that dominate a mean, and analysing ratios correctly.
CUPED, Winsorization, Delta method, Metric sensitivity and 2 more
Bayesian methods
Reasoning about effects as probability distributions rather than reject-or-not decisions: what that buys, what it costs, and what it does not fix.
Bayesian A/B testing, Prior distribution, Credible interval, Expected loss and 3 more
Geo testing methods
Measuring advertising by switching it on or off in whole markets — the design decisions, the estimation methods, and what each one assumes.
Geo experiment, Geo unit, Test and control markets, GeoLift and 6 more
Validity threats and corrections
The ways a technically correct test produces a wrong answer — peeking, multiplicity, broken randomisation — and the corrections that address each one.
Multiple comparisons, Sequential testing, Sample ratio mismatch and 11 more
Cause and effect
Methods for answering whether one thing caused another when you cannot randomise, and the biases that make an observed difference look like a causal one.
Causal inference, Counterfactual, Difference-in-differences, Confounding variable and 10 more
Media measurement and attribution
Whether your advertising caused anything, and the three families of method — geo experiments, mixed modelling and attribution — that claim to tell you.
Incrementality, Marketing mix modelling, Adstock, Diminishing returns and 9 more
Models and relationships
Regression and the diagnostics around it: what a coefficient claims, what R-squared does not, and the two ways a good-looking model misleads you.
Regression analysis, Correlation, R-squared, Overfitting and 1 more
Data and distributions
The statistical foundations everything else is built on: spread, the normal curve, why sample means behave, and what one extreme value can do to a test.
Standard deviation, Normal distribution, Central limit theorem, Outlier and 5 more

Frequently asked questions

Who is this glossary for?

Anyone who has to read, run or argue about an experiment without a statistics degree — product managers, growth and marketing teams, analysts early in their career, and engineers who have inherited an experimentation platform. Each entry assumes no prior notation and introduces any symbol it uses.

How is each entry structured?

A short definition first, so you can leave immediately if that is all you needed. Then a plain-English explanation, the formula where the term is a quantity, a worked example with real numbers, the common misconceptions, and links to the related terms and to any calculator that computes it.

Why are the formulas written without LaTeX?

Because a formula you can select and paste into a spreadsheet or a message to a colleague is more useful than one rendered as an image or a font you do not have. Everything here uses ordinary characters, so it copies as text.

A term I need is not here. Can you add it?

Probably, and we would like to know which one. The list is weighted towards experimentation, causal inference and marketing measurement because that is the work we do, so there are gaps in areas like time-series forecasting and survey methodology. Get in touch and tell us what you were looking for.

Can I link to or quote these definitions?

Yes. Link to the individual term page rather than this index — each one has a stable URL that will not change — and a citation back to it is appreciated but not required. Each entry also lists the textbooks and papers behind it if you need a primary source.

Knowing the words is not the same as knowing the answer.

Most measurement problems are not vocabulary problems. They are a metric that moves for the wrong reason, a test that never had the power to detect what you were looking for, or three channels each claiming the same conversion. If any of that sounds familiar, talk to us.

Statistics glossary

Browse by topic

Frequently asked questions

Knowing the words is not the same as knowing the answer.