Statistical significance gives bias a free pass

Whether or not "the foundations and the practice of statistics are in turmoil"1 , it is wiseto question methodswhose misuse has been lamented for over a century2-4 . Perhaps the most widespread misuse of statistics is takingthe crossingof some thresholdaslicense for declaring"statistical significance"and forgeneralizingfrom a single study. Such generalized conclusions are oftentaken up by science communicators, media, and political stakeholderswithout recognition of theiruncertainty. A major consequenceis flip-flopping headlines such as 'chocolate is good for you' followed by 'chocolate is bad for you'5 . No wonder only about athird of over 2000 respondents in a survey on the British public said they would trust data from medical trials6 .

[1]  David Gal,et al.  Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence , 2016, Manag. Sci..

[2]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[3]  When we make recommendations for scientific practice, we are (at best) acting as social scientists , 2019, European journal of clinical investigation.

[4]  Sander Greenland,et al.  Invited Commentary: The Need for Cognitive Science in Methodology , 2017, American journal of epidemiology.

[5]  John P. A. Ioannidis.  Retiring statistical significance would give bias a free pass , 2019, Nature.

[6]  Edwin G. Boring,et al.  Mathematical vs. scientific significance. , 1919 .

[7]  Sander Greenland,et al.  Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication , 2018, The American Statistician.

[8]  David M. Lane,et al.  Estimating effect size: Bias resulting from the significance criterion in editorial decisions , 1978 .

[9]  Brandy R. Maynard,et al.  Key concepts for making informed choices , 2019, Nature.

[10]  J. Carlin,et al.  Beyond Power Calculations , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[11]  Sander Greenland,et al.  Retire statistical significance , 2019 .

[12]  T. Hardwicke,et al.  Petitions in scientific argumentation: Dissecting the request to retire statistical significance , 2019, European journal of clinical investigation.

[13]  John P A Ioannidis,et al.  The Importance of Predefined Rules and Prespecified Statistical Analyses: Do Not Abandon Significance. , 2019, JAMA.

[14]  K. Pearson.  V. Note on the Significant or Non-significant character of a Sub-sample drawn from a Sample , 1906 .

[15]  Sander Greenland,et al.  Scientists rise up against statistical significance , 2019, Nature.

[16]  N. Lazar,et al.  Moving to a World Beyond “p < 0.05” , 2019, The American Statistician.

[17]  K. Fisher.  Statistical Tests , 1935, Nature.

[18]  David Gal,et al.  Abandon Statistical Significance , 2017, The American Statistician.

[19]  J. Lau,et al.  State of the evidence: current status and prospects of meta-analysis in infectious diseases. , 1999, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.