The New Statistics: Why and How

We need to make substantial changes to how we conduct research. First, in response to heightened concern that our published research literature is incomplete and untrustworthy, we need new requirements to ensure research integrity. These include prespecification of studies whenever possible, avoidance of selection and other inappropriate dataanalytic practices, complete reporting, and encouragement of replication. Second, in response to renewed recognition of the severe flaws of null-hypothesis significance testing (NHST), we need to shift from reliance on NHST to estimation and other preferred techniques. The new statistics refers to recommended practices, including estimation based on effect sizes, confidence intervals, and meta-analysis. The techniques are not new, but adopting them widely would be new for many researchers, as well as highly beneficial. This article explains why the new statistics are important and offers guidance for their use. It describes an eight-step new-statistics strategy for research with integrity, which starts with formulation of research questions in estimation terms, has no place for NHST, and is aimed at building a cumulative quantitative discipline.

[1]  David M Erceg-Hurn,et al.  Modern robust statistical methods: an easy way to maximize the accuracy and power of your research. , 2008, The American psychologist.

[2]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[3]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[4]  J. H. Steiger,et al.  Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. , 2004, Psychological methods.

[5]  Roger E. Kirk,et al.  The Importance of Effect Magnitude , 2008 .

[6]  Han L. J. van der Maas,et al.  Science Perspectives on Psychological an Agenda for Purely Confirmatory Research on Behalf Of: Association for Psychological Science , 2022 .

[7]  Robert Fletcher,et al.  Registration of clinical trials still moving ahead--September 2008 update to Uniform Requirements for Manuscripts Submitted to Biomedical Journals. , 2008, Croatian medical journal.

[8]  Kris N Kirby,et al.  BootES: An R package for bootstrap confidence intervals on effect sizes , 2013, Behavior research methods.

[9]  M. Masson.  Using confidence intervals for graphically based data interpretation. , 2003, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[10]  Geoff Cumming,et al.  Confidence intervals and replication: where will the next mean fall? , 2006, Psychological methods.

[11]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[12]  Carla Hansen,et al.  More Tools for the Synthesist’s Toolbag in Harris Cooper’s Research Synthesis and Meta-Analysis: A Step-by-Step Approach (4th ed.) , 2009 .

[13]  Klaus Fiedler,et al.  The Long Way From α-Error Control to Validity Proper , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[14]  Fiona Fidler,et al.  The statistical recommendations of the American Psychological Association Publication Manual: Effect sizes, confidence intervals, and meta‐analysis , 2012 .

[15]  Geoff Cumming,et al.  Inference by eye: Reading the overlap of independent confidence intervals , 2009, Statistics in medicine.

[16]  Paul D. Ellis,et al.  The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results , 2010 .

[17]  Effect Size Estimation and Confidence Intervals , 2012 .

[18]  Geoff Cumming,et al.  Inference by Eye: Pictures of Confidence Intervals and Thinking About Levels of Confidence , 2007 .

[19]  D. G. Wastell,et al.  Statistics with confidence—Confidence intervals and statistical guidelines , 1991 .

[20]  G. Loftus,et al.  Why Figures with Error Bars Should Replace p Values Some Conceptual Arguments and Empirical Demonstrations , 2015 .

[21]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[22]  C. Borland,et al.  Effect Size , 2019, SAGE Research Methods Foundations.

[23]  G. Cumming,et al.  Reform of statistical inference in psychology: The case ofMemory & Cognition , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[24]  R. Grissom,et al.  Effect Sizes for Research : Univariate and Multivariate Applications, Second Edition , 2005 .

[25]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[26]  Kees van Deemter.  Not Exactly: In Praise of Vagueness , 2010 .

[27]  A. Beck,et al.  Comparison of Beck Depression Inventories -IA and -II in psychiatric outpatients. , 1996, Journal of personality assessment.

[28]  A. Riopelle,et al.  On confidence intervals for within-subjects designs. , 2005, Psychological methods.

[29]  Geoff Cumming.  META-ANALYSIS: PICTURES THAT EXPLAIN HOW EXPERIMENTAL FINDINGS CAN BE INTEGRATED , 2006 .

[30]  Ken Kelley,et al.  Methods for the Behavioral, Educational, and Social Sciences: An R package , 2007, Behavior research methods.

[31]  G. Newman,et al.  CONFIDENCE INTERVALS , 1987, The Lancet.

[32]  Rex B. Kline,et al.  Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research , 2004 .

[33]  D. Vaux,et al.  Error bars in experimental biology , 2007, The Journal of cell biology.

[34]  C. Sansone,et al.  Improving the Dependability of Research in Personality and Social Psychology , 2014, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[35]  Jennifer J. Richler,et al.  Effect size estimates: current use, calculations, and interpretation. , 2012, Journal of experimental psychology. General.

[36]  Harris Cooper,et al.  Research Synthesis and Meta-Analysis: A Step-by-Step Approach , 2009 .

[37]  G. Cumming,et al.  Replication and Researchers' Understanding of Confidence Intervals and Standard Error Bars. , 2004 .

[38]  Rex B. Kline,et al.  Beyond Significance Testing: Statistics Reform in the Behavioral Sciences , 2013 .

[39]  A. Jansen.  Bayesian Methods for Ecology , 2009 .

[40]  Janet L. Johnson,et al.  Theory Testing Using Quantitative Predictions of Effect Size. , 2008, Applied psychology = Psychologie appliquee.

[41]  H. Cooper.  Reporting Research in Psychology: How to Meet Journal Article Reporting Standards , 2010 .

[42]  S Greenland,et al.  The fallacy of employing standardized regression coefficients and correlations as measures of effect. , 1986, American journal of epidemiology.

[43]  Leif D. Nelson,et al.  Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” , 2014 .

[44]  E. Masicampo,et al.  A peculiar prevalence of p values just below .05 , 2012, Quarterly journal of experimental psychology.

[45]  G. Cumming,et al.  Inference by eye: confidence intervals and how to read pictures of data. , 2005, The American psychologist.

[46]  G. Cumming,et al.  Confidence Intervals Permit, but Do Not Guarantee, Better Inference than Statistical Significance Testing , 2010, Front. Psychology.

[47]  In-Sue Oh,et al.  Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. , 2009, The British journal of mathematical and statistical psychology.

[48]  B. Spellman.  Introduction to the Special Section on Research Practices , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[49]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[50]  G. Cumming.  Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better , 2008, Perspectives on psychological science : a journal of the Association for Psychological Science.

[51]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[52]  J. Kruschke.  Doing Bayesian Data Analysis: A Tutorial with R and BUGS , 2010 .

[53]  R. J. Boik.  Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach , 2001 .

[54]  Stacey A. Hancock.  Modern Statistics for the Social and Behavioral Sciences: A Practical Introduction , 2012 .

[55]  G. Cumming,et al.  Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think , 2004, Psychological science.

[56]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[57]  G. Cumming,et al.  Confidence intervals : better answers to better questions. , 2009 .

[58]  G. Cumming.  Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[59]  J. Rodgers.  The epistemology of mathematical and statistical modeling: a quiet methodological revolution. , 2010, The American psychologist.

[60]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[61]  G. Cumming,et al.  The value of RCT evidence depends on the quality of statistical analysis. , 2008, Behaviour research and therapy.

[62]  G. Cumming,et al.  A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions , 2001 .

[63]  Sue Finch,et al.  Putting research in context: understanding confidence intervals from one or more studies. , 2009, Journal of pediatric psychology.

[64]  J. Hoenig,et al.  Statistical Practice The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis , 2001 .

[65]  Douglas G Bonett,et al.  Meta-analytic interval estimation for standardized and unstandardized mean differences. , 2009, Psychological methods.

[66]  Harris Cooper,et al.  The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. , 2009, Psychological methods.

[67]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[68]  Heather Douglas,et al.  Rejecting the Ideal of Value‐Free Science , 2007 .

[69]  Thom Baguley,et al.  Serious stats: a guide to advanced statistics for the behavioral sciences , 2012 .