StatReason

P Value Misconception Checker

Read each statement below and check whether you would call it safe, unsafe, or incomplete before revealing the classification and the reason behind it.

Want the calculator? P-Value Calculator

Export to PDF

What this answers

This tool answers "is this specific way of phrasing a p-value result safe to say?" Rather than repeating abstract rules, it walks through six concrete statements, the kind that actually appear in reports and conversations, and classifies each one with a specific reason.

The three classifications

A statement is safe if it accurately describes what the p-value measures without overclaiming. A statement is unsafe if it makes a claim the p-value cannot support, such as a probability statement about the null hypothesis itself or a claim about practical importance. A statement is incomplete if it is not technically false but leaves out a qualifier that changes how it should be read, such as omitting the specific null model or the selected alpha level.

Worked example

"The p-value is 0.03, so there is a 3 percent chance the null hypothesis is true" is unsafe: a p-value is a probability about data under the null model, not a probability assigned to the null model itself. "The p-value is 0.03, which is below our alpha of 0.05" is safe: it states exactly what was compared and to what threshold. "The result was significant" is incomplete: it does not state the alpha level or the specific test performed, both of which change how a reader should weigh the claim.

Assumption audit

Calculated from your data: nothing; this tool checks fixed statements against a fixed classification, not any data you enter.
Evidence to review: the specific reason given for each classification, which explains the general rule you can apply to a new statement not listed here.
You must verify: that any p-value statement you write or read yourself follows the same reasoning applied to these six examples.

Source

These classifications follow the American Statistical Association's statement on statistical significance and p-values and the shared statistical reasoning contract every StatReason engine is built against.

Limitations

This tool covers six representative statements, not every possible phrasing; use the general reasoning behind each classification to evaluate statements not listed here.

Next action: read the full explanation at What a P Value Does and Does Not Mean, then check your own result with the P-Value Calculator, the Independent t-Test, or the One-Way ANOVA Calculator.