Justify your alpha

In response to recommendations to redefine statistical significance to P ≤ 0.005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Nicholas P. Holmes | Andrea E. Martin | Junpeng Lao | Gerit Pfuhl | Emily S. Cross | Henrik Danielsson | Christer Johansson | Rolf A. Zwaan | Casper J. Albers | Iris K. Schneider | Daniel Lakens | Fred Hasselman | Deborah G. Mayo | Thom Baguley | Christopher R. Madan | Jeffrey R. Spies | James A. Grange | Tim Smits | Miguel A. Vadillo | Brian D. Earp | Randy J. McCarthy | Aaron R. Caldwell | Robert Guggenberger | Sau-Chin Chen | Rickard Carlsson | Barbara Konat | Kevin McConway | Tal Yarkoni | Peder M. Isager | Neil Stenhouse | Michael Ingre | Gustav Nilsonne | James G. Field | Michele I. Feist | Erin M. Buchanan | Monica Gonzalez-Marquez | Lisa DeBruine | Zsuzsika Sjoerds | Lincoln J. Colling | David A. Kenny | Ignazio Ziano | Stephen D. Benning | Stephen R. Martin | Matthew A. J. Apps | Crystal N. Steltenpohl | Raymond B. Becker | Vishnu Sreekumar | Manojkumar Selvaraju | Marcel A. L. M. Assen | David Manheim | Jiří Lukavský | Ahmed A. Khalil | Farid Anvari | Gerine M. A. Lodder | Daniel J. Dunleavy | Donald R. Williams | Hanna K. Isotalus | D. Mayo | N. Holmes | T. Baguley | K. McConway | J. Grange | F. Hasselman | D. Lakens | G. Nilsonne | M. V. Assen | T. Yarkoni | M. Ingre | L. DeBruine | M. Apps | S. Argamon | E. Buchanan | A. Grieve | Z. Sjoerds | S. Parsons | R. Becker | C. Albers | C. Madan | P. Isager | Rickard Carlsson | H. Danielsson | J. Lukavský | M. Gonzalez-Marquez | B. Calster | G. Lodder | J. O. Xivry | Matt N. Williams | Andrea E. Martin | Anne-Laura Harmelen | T. Smits | A. Nio | N. Fox | G. Pfuhl | Sau-chin Chen | R. Guggenberger | D. Kenny | John J. Sakon | H. Isotalus | D. Manheim | B. Earp | I. Schneider | Junpeng Lao | Wojciech Świątkowski | Vishnu Sreekumar | M. Selvaraju | Daniel E. Bradford | Christer Johansson | S. A. Saribay | Neil Stenhouse | Federico G. Adolfi | Shlomo E. Argamon | Ben Calster | Bryan Chung | Gary S. Collins | Zander Crook | Sameera Daniels | Jason D. Ferrell | Nicholas W. Fox | Amanda Friesen | Caio Gomes | Andrew P. Grieve | James Grist | Anne-Laura Harmelen | Kevin D. Hochard | Mark R. Hoffarth | Konrad Juszczyk | Erik Gahner Larsen | Colin McFarland | Amanda Q. X. Nio | Cilene Lino Oliveira | Jean-Jacques Orban Xivry | Sam Parsons | Kimberly A. Quinn | S. Adil Saribay | Samuel G. Smith | Wojciech Świątkowski | Samantha E. Williams | M. Hoffarth | Jason Ferrell | A. Khalil | Farid Anvari | Konrad Juszczyk | Amanda Friesen | E. Larsen | J. Grist | I. Ziano | A. Martin | G. Collins | Stephen R Martin | C. Gomes | Barbara Konat | K. Quinn | K. Hochard | D. Dunleavy | Bryan Chung | Z. Crook | Sameera Daniels | Colin McFarland | Cilene L Oliveira | Samantha E Williams | Caio Gomes | Andrea E. Martin | M. Vadillo

[1]  L. Hothorn.  Statistical issues in drug development (2nd edn). Stephen Senn, John Wiley & Sons Ltd, Chichester, 2007. No. of pages: xix + 498. Price: $130.00. ISBN 978‐0‐470‐01877‐4 , 2009 .

[2]  Pia Rotshtein,et al.  Registered Reports: Realigning incentives in scientific publishing , 2015, Cortex.

[3]  Nathaniel Rothman,et al.  Assessing the Probability That a Positive Report is False: An Approach for Molecular Epidemiology Studies , 2004 .

[4]  S. Gad.  Pharmaceutical Statistics , 2003 .

[5]  Günter Altner,et al.  The Nature of human behaviour , 1976 .

[6]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[7]  Louis Lyons,et al.  Discovering the Significance of 5 sigma , 2013, 1310.1284.

[8]  Joseph F. Mudge,et al.  Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests , 2012, PloS one.

[9]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[10]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[11]  Deborah G. Mayo,et al.  Statistical Inference as Severe Testing , 2018 .

[12]  Hugh P. Possingham,et al.  Minimizing the cost of environmental management decisions by optimizing statistical thresholds , 2004 .

[13]  Valen E. Johnson,et al.  On the Reproducibility of Psychological Science , 2017, Journal of the American Statistical Association.

[14]  Andrew P Grieve,et al.  How to test hypotheses if you must , 2015, Pharmaceutical statistics.