Chapter 8. Abandon statistical significance

This chapter may come as a surprise to you.  For a very long time, one of the first ideas that biology students learn in biostatistics is that p < 0.05 is a “significant” result and that p > 0.05 is a “non-significant” result.  I remember learning this idea during my undergraduate degree (although I had no understanding of what a p-value really was at that time!). 

This idea of ‘significance’ is commonly ingrained in interpretation of statistical tests in biology.  But, it turns out that this approach leads to serious errors.  In fact, the American Statistics Association declared in 2019 that researchers should immediately stop interpreting their results in terms of “statistical significance”, and never use the term “statistically significant” (and all variants thereof) ever again.  You can read more about this perspective and alternative methods to interpret results in the 73rd volume of the American Statistician. 

View online articles in the American Statistician, Volume 73.

This chapter explores the arguments to abandon the concept of statistical significance, and recommends approaches to interpret results.  First, we recommend that p-values be interpreted along a continuum, where very small values provide strong evidence to reject a null hypothesis and larger p-values provide weaker evidence against the null hypothesis. 

Following results from Benjamin et al. (2018; see below), we suggest that p-values around 0.005 or smaller present ‘substantial’ to ‘strong’ evidence to reject the null hypothesis; we also recommend that p-values in the vicinity of 0.05 be termed ‘moderate’ evidence against the null hypothesis (but see, below), and that larger p-values represent ‘weak’ evidence against the null hypothesis.  Second, we highlight estimates of effect size (with 95% confidence intervals) as a biologically informed approach to understand one’s results.

Please note:  I suggest in these videos that we consider p-values around 0.05 to be termed 'moderate' evidence for an effect.  In the following article, Benjamin et al. (2018; redefine statistical significance. Nature Human Behaviour) indicates that the 0.05 represents 'weak' evidence for an effect and justify their argument in three ways:

We used the term 'moderate' for two reasons.  First, we fear that declaring 0.05 as 'weak' evidence for an effect  (even if it is true) will prove too radical for some people and hinder a transition away from the concept of 'statistical significance'.  Second the terminology “moderate” follows precedent elsewhere.  The article by Muff et al. (2022), listed in the recommended reading below, discusses this topic and provides excellent advice on how to report results in scientific publications in a modern approach.