Almost every statistics course deducts marks for the same sentence: the results prove there is no difference. Here is what a p-value actually says, what it never says, and how to write the result up so it scores.
What a p-value actually measures
A p-value is the probability of getting data at least as extreme as yours, assuming the null hypothesis is true. That is the whole definition. It is a statement about data under an assumption, not a statement about whether the assumption is correct. It does not tell you the probability that your hypothesis is true, and it does not tell you how big or important the effect is.
The three sentences that cost marks
First: the result proves there is no effect. A nonsignificant p-value means you failed to reject the null, which is not the same as confirming it; your study may simply have been underpowered. Second: p = .04 means there is a 4 percent chance the result is due to chance. It does not. Third: p = .001 means the effect is large. It does not; with 5,000 participants a trivial difference will produce a tiny p-value.
Always report the effect size next to it
Significance tells you whether an effect is detectable. Effect size tells you whether it matters. For t-tests report Cohen's d, where roughly 0.2 is small, 0.5 medium and 0.8 large. For ANOVA report eta squared or partial eta squared. For chi-square report Cramer's V. For regression report R squared and standardised betas. US rubrics increasingly award points for effect size specifically.
Confidence intervals do the work p-values cannot
A 95 percent confidence interval gives you the range of effects your data are compatible with. A mean difference of 6.5 points with an interval of [1.6, 11.4] tells your reader something useful. The same difference with an interval of [-0.4, 13.4] tells them the data are also compatible with no difference at all. Report the interval every time.
The APA 7 reporting template
Write the test, degrees of freedom, statistic, exact p-value, confidence interval and effect size in one sentence, then one sentence of plain-English meaning. For example: students in the workshop group scored higher than those who did not attend, t(60) = 2.63, p = .011, 95% CI [1.56, 11.44], d = 0.67. Use exact p-values rather than p < .05, and report p < .001 only when the value is smaller than that.
A note on p-hacking
Running every test until one comes out significant, dropping inconvenient cases, or deciding the hypothesis after seeing the data will eventually produce a significant result by chance alone. Pre-state your hypotheses, report every test you ran, and correct for multiple comparisons with Bonferroni or Holm when you run a family of tests. Markers notice, and so do committees.
Key takeaways
- → A p-value is about the data under the null, not about your hypothesis
- → Nonsignificant means failed to reject, not proven equal
- → Always report an effect size next to the p-value
- → Confidence intervals tell readers what the data are compatible with
- → Use exact p-values in APA 7 and correct for multiple comparisons