StatReason

Statistical Significance vs Practical Significance

Fix an effect size and change only the sample size below to see the p-value fall while the effect itself never changes size.

Want the calculator? P-Value Calculator

Export to PDF

0.3 units, held constant

10 per group

Fixed effect size
Approximate two-sided p-value
Significant at alpha .05?
Check your own effect size directly: Effect Size and Why Magnitude Matters

What this answers

This visual answers "can a tiny, practically unimportant difference still produce a small p-value?" It holds the raw effect size fixed at whatever value you set and changes only the sample size, so you can watch statistical significance appear on a difference that never grew in size.

Why sample size alone can create significance

A p-value depends on both the size of an effect and the precision of the estimate. Precision improves as sample size grows, because the standard error of an estimate shrinks proportionally to the square root of the sample size. A fixed, small effect that is not statistically distinguishable from zero at a small sample size can become statistically significant at alpha .05 once the sample is large enough, purely because the standard error has shrunk far enough to detect it.

Worked example

With an effect size of 0.3 units and 10 observations per group, the p-value is typically well above .05. Raise the sample size toward 5,000 per group while leaving the effect size at 0.3, and the p-value falls below .05, often far below it, even though the difference itself is exactly as small as it was before. The finding becomes statistically significant without becoming more important in practice.

Assumption audit

Calculated from your data: in this visual, an approximate p-value from your chosen effect size and sample size under a simplified two-sample comparison; a real test calculator computes this exactly from your actual data.
Evidence to review: whether the fixed effect size you are testing against would matter in your specific decision, independent of whatever p-value a large sample produces.
You must verify: what magnitude of difference is practically meaningful in your context before running the test, since no p-value, however small, can supply that judgment for you.

Source

This distinction follows the significance-versus-magnitude guidance in the American Statistical Association's statement on statistical significance and the shared statistical reasoning contract every StatReason engine is built against.

Limitations

This visual uses a simplified two-sample approximation to illustrate the relationship; it is not a substitute for running the actual test on your data. Use the linked engines for an exact result and its own confidence interval.

Next action: run the exact test with the P-Value Calculator, plan a study with adequate power using the Sample Size Calculator, or check a standardized effect size with the Independent t-Test.