Life After NHST: How to Describe Your Data Without “p-ing” Everywhere

In this article we provide concrete guidance to researchers on ways that they can explore and communicate the results of their studies. Although we believe the methods we outline are important for any study, they are particularly useful for researchers who wish to avoid the null hypothesis significance testing paradigm. We articulate three basic principles of data presentation: (a) use graphic displays to facilitate understanding of descriptive statistics, (b) provide measures of variability with measures of central tendency for continuous outcomes, and (c) compute and thoughtfully interpret effect sizes and effect size translations. We then put these principles into action using data drawn from two real social psychological experiments and provide tools (including software code and a new effect size translation) that will help researchers to quickly and efficiently adopt the recommendations that they find sensible.

[1]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[2]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[3]  Donald B. Rubin,et al.  A Note on Percent Variance Explained as A Measure of the Importance of Effects , 1979 .

[4]  J. Fleiss,et al.  Significance tests have a role in epidemiologic research: reactions to A. M. Walker. , 1986, American journal of public health.

[5]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[6]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[7]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[8]  K. McGraw,et al.  A common language effect size statistic. , 1992 .

[9]  On P values and confidence intervals (why can't we P with more confidence?) , 1993, Clinical chemistry.

[10]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[11]  W. Mischel,et al.  A cognitive-affective system theory of personality: reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. , 1995, Psychological review.

[12]  J. Baron,et al.  Confusion of Relative and Absolute Risk in Valuation , 1997 .

[13]  Robert P. Abelson,et al.  On the Surprising Longevity of Flogged Horses: Why There Is a Case for the Significance Test , 1997 .

[14]  R. L. Hagen.  In praise of the null hypothesis statistical test. , 1997 .

[15]  K J Rothman,et al.  That confounded P-value. , 1998, Epidemiology.

[16]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[17]  Why Research on Science and Mathematics Education in Rural Schools Is Important or the Mean Is the Wrong Message. , 2001 .

[18]  R. Viertl.  On the Future of Data Analysis , 2002 .

[19]  D. Trafimow.  Hypothesis testing and theory evaluation at the boundaries: surprising insights from Bayes's theorem. , 2003, Psychological review.

[20]  G. Gigerenzer,et al.  Simple tools for understanding risks: from innumeracy to insight , 2003, BMJ : British Medical Journal.

[21]  G. Cumming,et al.  Editors Can Lead Researchers to Confidence Intervals, but Can't Make Them Think , 2004, Psychological science.

[22]  P. Ubel,et al.  Reducing the Influence of Anecdotal Reasoning on People’s Health Care Decisions: Is a Picture Worth a Thousand Statistics? , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[23]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[24]  A. Gelman,et al.  The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant , 2006 .

[25]  Peter C Gøtzsche,et al.  Believability of relative risks and odds ratios in abstracts: cross sectional study , 2006, BMJ : British Medical Journal.

[26]  R. Boruch.  The null hypothesis is not called that for nothing: statistical tests in randomized trials , 2007 .

[27]  Judith Covey,et al.  A Meta-analysis of the Effects of Presenting Treatment Benefits in Different Formats , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[28]  Wolfgang Viechtbauer,et al.  Publication bias in meta-analysis: Prevention, assessment and adjustments , 2007, Psychometrika.

[29]  Harris Cooper,et al.  Evaluating and Interpreting Research Syntheses in Adult Learning and Literacy. NCSALL Occasional Paper. , 2007 .

[30]  G. Cumming.  Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better , 2008, Perspectives on psychological science : a journal of the Association for Psychological Science.

[31]  Kay Dickersin,et al.  Outcome reporting in industry-sponsored trials of gabapentin for off-label use. , 2009, The New England journal of medicine.

[32]  P. Andersen,et al.  Image Induction and Social Influence: Explication and Initial Tests , 2009 .

[33]  Shiloh E. Beckerley,et al.  Effects of Media Images on Attitudes Toward Tanning , 2010 .

[34]  J. Valentine,et al.  Outcome-Reporting Bias in Education Research , 2013 .

[35]  On teaching about the coefficient of variation in introductory statistics courses , 2014 .

[36]  G. Cumming.  The New Statistics: Why and How , 2013 .

[37]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[38]  David Trafimow.  The intelligibility of r or r2 as an effect size statistic: dichotomous variables , 2015, Front. Psychol..

[39]  Lisa L. Harlow,et al.  Eight Common but False Objections to the Discontinuation of Significance Testing in the Analysis of Research Data , 2016 .

[40]  Yoshinori Hatori,et al.  The Difference between Significant and Non-significant , 2017 .