StatReason

Permutation Test Calculator

Test whether two independent groups genuinely differ, without assuming any particular distribution shape, by directly asking how extreme your observed difference is compared to every way the group labels could have been shuffled.

What this answers

This test answers "how unusual is my observed difference between groups, if group membership genuinely had no effect and the labels were exchangeable?" without assuming normality, equal variance, or any other distributional shape a t-test requires.

How it is calculated

All observations from both groups are pooled together, then split into new "Group A" and "Group B" sized exactly like the originals, and the chosen statistic is recomputed on each such split. When the total number of possible splits is small enough, every single one is enumerated exactly; otherwise, this calculator draws 9,999 random splits using a seeded, fully reproducible generator. The p-value is the fraction of splits (exact or random) that produce a difference at least as extreme as the one actually observed.

Worked example

For Group A of 1, 2, 3 and Group B of 4, 5, 6, there are only 20 total ways to split 6 values into two groups of 3, few enough to enumerate exactly. Since this data shows the most extreme possible separation (every A value is below every B value), only 2 of the 20 splits are at least this extreme, giving an exact p-value of exactly .1.

Assumption audit

Calculated from your data: the observed statistic, whether exact enumeration was feasible given your sample sizes, the number of permutations actually used, and the resulting p-value.
Evidence to review: whether your two groups look similarly shaped aside from the difference you are testing; a permutation test on means still answers a question about means specifically, even though it makes no distributional assumption to get there.
You must verify: exchangeability, that under the null hypothesis every observation could equally plausibly have landed in either group; this is the core assumption a permutation test relies on and cannot check from the data alone.

What this result does not mean

A significant permutation test means the observed group difference is unlikely under random relabeling, not that group assignment caused the difference, unless the groups were themselves randomly assigned by design.

Limitations

This calculator supports two independent groups only, comparing a difference of means or medians; paired designs and correlation-based permutation tests are different designs not covered by this specific engine.