Wilcoxon test calculator

Reach for a rank test in two situations. The first is an ordinal outcome — a 1-to-5 rating, an NPS bucket, a severity grade — where the numbers label an ordering and their arithmetic mean is not a quantity that exists. The second is a continuous outcome badly enough behaved, at a sample small enough, that the t-test's reliance on the Central Limit Theorem is a hope rather than a fact. Use rank-sum when the samples are independent and signed-rank when they are pairs, as you would choose between the two-sample and the paired t-test.

Be precise about what the rank-sum test tests, because the standard description of it is wrong. Its null is that a randomly chosen observation from one group is equally likely to be larger or smaller than one from the other — P(X > Y) = 0.5, stochastic equality. That becomes a statement about medians only under the extra assumption that the two distributions have the same shape and differ by a shift. Without it the test can return a decisive p-value with the medians identical, because a difference in spread or skew is enough to make one group stochastically larger. Mann and Whitney's 1947 title says "stochastically larger", which is exactly what "compares medians" throws away.

Use something else when the estimand matters more than the robustness. If your decision depends on total revenue, ordering is not enough: a rank test can say treatment users tend to spend more without saying whether the business makes any money. There, a t-test on a winsorised outcome or a bootstrap interval is better — see our note on outliers. Three or more groups want Kruskal–Wallis rather than one-way ANOVA; binary outcomes want a proportions test; paired binary outcomes want McNemar's; Spearman's rho lives on the correlation page.

"Non-parametric" means no assumed distributional family. It does not mean assumption-free, and the assumptions that remain are the ones that bite:

• Independence — between the two samples for rank-sum, across pairs for signed-rank. Ranking does nothing about clustering, repeated measures from the same user, or a unit that appears in both arms.
• For the shift interpretation only: the two distributions have the same shape. Report "the treatment group is about 12 seconds slower" and you are assuming it. Report "a treatment user is slower than a control user 62% of the time" and you are not. Say which claim you are making.
• For signed-rank: the paired differences are symmetric about their centre. Rarely stated and often false, since differences from a skewed outcome tend to be skewed too. Under asymmetry the test is still valid, but as a test of the pseudo-median — the Hodges–Lehmann centre — not the median of the differences.
• Ties are handled but not free. The exact null distribution assumes none; with ties the calculator switches to the tie-corrected normal approximation. On a five-point scale almost everything is tied, and an ordinal model will tell you more than any p-value.
• The choice of test was not made by looking at the data. Running a normality test and then picking between the t-test and this one is a two-step procedure with a worse combined error rate than either used unconditionally, because the choice is itself an inference nothing downstream accounts for. Decide from the metric's known shape before collection, like the rest of the analysis plan.

You are comparing time-to-first-action between two onboarding flows. The distribution is heavily right-skewed — most users finish inside a minute, a tail wander off and come back — so you would rather not stake the analysis on means. You state the smallest worthwhile effect on the test's own scale: a new-flow user beats an old-flow user 60% of the time.

The two routes landing within a couple of percent of each other is the point of running both: the ARE adjustment is doing its job and you have not specified a much larger effect than you meant to. And 0.60 is smaller than it sounds — complete overlap is 0.50, so this is a fifth of the way from no difference to the new flow always winning. What the test will not give you is a duration. If the roadmap needs "14 seconds faster on average", that has to come from a Hodges–Lehmann shift estimate or a mean with a bootstrap interval.

Lead with the effect size, because a rank statistic is unreadable on its own. U = 41,208 means nothing to anybody; "a randomly chosen treatment user scores higher than a control user 61% of the time" is immediately usable and is a direct statement about the quantity the test is built on. Where the equal-shape assumption is defensible, add the Hodges–Lehmann estimate — the median of all cross-group differences — with its interval, which puts the result back in original units.

Then be careful about what significance licenses. A small p-value says the two samples are unlikely to have come from the same distribution. It does not say the medians differ, or by how much: if the treatment group is both slightly better and noticeably more variable, the test can be significant with the medians sitting on top of each other. Plot the two distributions before writing the conclusion — a habit worth keeping whether or not the data is well behaved.

Watch the boundary between the exact test and the approximation. Below 50 per group the p-value is exact and the discreteness is real — at very small samples no arrangement of the ranks may produce a p below 0.05, which is a property of the design rather than a finding. Above it the tie-corrected approximation is accurate, but heavy tying degrades the whole approach.

A non-significant rank test is not evidence the two distributions match, and the remedy is awkward: equivalence testing on ranks is not standard practice. The pragmatic route is to define the margin on a scale you can defend — a difference in means, or a bound on P(X > Y) — and run an equivalence test or a non-inferiority test, stating the change of estimand openly.