Replicability Crisis in Social Psychology: Looking at the Past to Find New Pathways for the Future

Over the last few years, psychology researchers have become increasingly preoccupied with the question of whether findings from psychological studies are generally replicable. The debates have originated from some unfortunate events of scientific misconduct in the field, and they have reached a climax with the recent discovery of a relatively weak rate of replicability of published literature, leading to the so-called replicability crisis in psychology.  The present paper is concerned with examining the issue of replicability in the field of social psychology. We begin by drawing a state of the art of the crisis in this field. We then highlight some possible causes for the crisis, discussing topics of statistical power, questionable research practices, publication standards, and hidden auxiliary assumptions of context-dependency of social psychological theories. Finally, we argue that given the absence of absolute falsification in science, social psychology could greatly benefit from adopting McGuire’s perspectivist approach to knowledge construction.

[1]  Alan Chalmers,et al.  What is this thing called science? - an assessment of the nature and status of science and its methods (2. ed.) , 1977, UQP nonfiction.

[2]  J. Wicherts,et al.  The (mis)reporting of statistical results in psychology journals , 2011, Behavior research methods.

[3]  Brian A. Nosek,et al.  An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  P. Meehl Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[5]  Rink Hoekstra,et al.  Probability as certainty: Dichotomous thinking and the misuse ofp values , 2006, Psychonomic bulletin & review.

[6]  William J. McGuire,et al.  Psychology of science: A perspectivist approach to the strategic planning of programmatic scientific research , 1989 .

[7]  E. Wagenmakers A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[8]  D. Lakens,et al.  Sailing From the Seas of Chaos Into the Corridor of Stability , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[9]  W. Mcguire Creative hypothesis generating in psychology: some useful heuristics. , 1997, Annual review of psychology.

[10]  J. Brooks Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[11]  G. Cumming,et al.  Confidence Intervals Permit, but Do Not Guarantee, Better Inference than Statistical Significance Testing , 2010, Front. Psychology.

[12]  H. Markus,et al.  Culture and the self: Implications for cognition, emotion, and motivation. , 1991 .

[13]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[14]  Eric-Jan Wagenmakers,et al.  Editors’ introduction to the special issue “Bayes factors for testing hypotheses in psychological research : Practical relevance and new developments” , 2016 .

[15]  G. Cumming Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis , 2011 .

[16]  J. Crocker,et al.  Addressing Scientific Fraud , 2011, Science.

[17]  N. Kerr HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[18]  S. Fiske,et al.  Mind the Gap: In Praise of Informal Sources of Formal Theory , 2004, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[19]  Jacob Cohen The earth is round (p < .05) , 1994 .

[20]  U. Schimmack The ironic effect of significant results on the credibility of multiple-study articles. , 2012, Psychological methods.

[21]  Imre Lakatos,et al.  The Methodology of Scientific Research Programmes , 1978 .

[22]  V. Johnson Revised standards for statistical evidence , 2013, Proceedings of the National Academy of Sciences.

[23]  J. Jost,et al.  The Estrangement of Social Constructionism and Experimental Social Psychology: History of the Rift and Prospects for Reconciliation , 2002 .

[24]  Leif D. Nelson,et al.  Correcting the past: failures to replicate ψ. , 2012, Journal of personality and social psychology.

[25]  K. Gergen Social Psychology as History , 1973 .

[26]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[27]  O. Klein Constructing social psychology. Creative and critical processes , 2001 .

[28]  Z. Dienes Bayesian Versus Orthodox Statistics: Which Side Are You On? , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[29]  S. Schmidt Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences , 2009 .

[30]  C. F. Bond,et al.  One Hundred Years of Social Psychology Quantitatively Described , 2003 .

[31]  Yoel Inbar,et al.  Association between contextual dependence and replicability in psychology may be spurious , 2016, Proceedings of the National Academy of Sciences.

[32]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[33]  Caesar Saloma,et al.  Things I have learned so far , 2008 .

[34]  Matthew C. Makel,et al.  Replications in Psychology Research , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[35]  G. Gigerenzer,et al.  Do studies of statistical power have an effect on the power of studies , 1989 .

[36]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[37]  G. Gigerenzer Mindless statistics , 2004 .

[38]  J. Bargh,et al.  Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action. , 1996, Journal of personality and social psychology.

[39]  M. Inzlicht,et al.  A Multilab Preregistered Replication of the Ego-Depletion Effect , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[40]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[41]  M. Hagberg Editorial , 2004 .

[42]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[43]  W. Mcguire,et al.  A Perspectivist Approach to Theory Construction , 2004, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[44]  Daniele Fanelli,et al.  Negative results are disappearing from most disciplines and countries , 2011, Scientometrics.

[45]  M. Brewer,et al.  Research Design and Issues of Validity , 2000 .

[46]  E. Garfield The history and meaning of the journal impact factor. , 2006, JAMA.

[47]  James W. Neuliep,et al.  Reviewer bias against replication research. , 1993 .

[48]  R. Baumeister,et al.  Ego depletion: is the active self a limited resource? , 1998, Journal of personality and social psychology.

[49]  J. H. Steiger,et al.  The Problem Is Epistemology, Not Statistics: Replace Significance Tests by Confidence Intervals and Quantify Accuracy of Risky Numerical Predictions , 2002 .

[50]  Jay J Van Bavel,et al.  Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology , 2016, Proceedings of the National Academy of Sciences.

[51]  Patrizio Emanuele Tressoldi Replication Unreliability in Psychology: Elusive Phenomena or “Elusive” Statistical Power? , 2012, Front. Psychology.

[52]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[53]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[54]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[55]  Axel Cleeremans,et al.  Behavioral Priming: It's All in the Mind, but Whose Mind? , 2012, PloS one.

[56]  R. Giner-Sorolla,et al.  Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[57]  J. Rossi,et al.  Statistical power of psychological research: what have we gained in 20 years? , 1990, Journal of consulting and clinical psychology.

[58]  G. Cumming,et al.  The New Statistics , 2014, Psychological science.

[59]  P. Meehl Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It , 1990 .

[60]  Mahzarin R. Banaji,et al.  Perspectivism in Social Psychology: The Yin and Yang of Scientific Progress , 2004 .

[61]  D. Bem Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. , 2011, Journal of personality and social psychology.

[62]  Daryl J. Bem,et al.  Writing the Empirical Journal Article , 2021, The Compleat Academic.

[63]  Arthur A. Stukas,et al.  Interpreting effect sizes: Toward a quantitative cumulative social psychology , 2014 .

[64]  Gerd Gigerenzer,et al.  Surrogate Science , 2015 .

[65]  C. Ferguson,et al.  A Vast Graveyard of Undead Theories , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[66]  M. Lee,et al.  Statistical Evidence in Experimental Psychology , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[67]  E. Wagenmakers,et al.  Detecting and avoiding likely false‐positive findings – a practical guide , 2017, Biological reviews of the Cambridge Philosophical Society.

[68]  S. Heine,et al.  Psychological universals: what are they and how can we know? , 2005, Psychological bulletin.

[69]  J. Vandekerckhove,et al.  A Bayesian Perspective on the Reproducibility Project: Psychology , 2016, PloS one.

[70]  G. Keppel Design and analysis: A researcher's handbook, 3rd ed. , 1991 .

[71]  G. Hofstede Culture′s Consequences: Comparing Values, Behaviors, Institutions and Organizations Across Nations , 2001 .

[72]  Z. Dienes How Bayes factors change scientific practice , 2016 .

[73]  Ana B. Chica,et al.  Attentional Routes to Conscious Perception , 2012, Front. Psychology.

[74]  Axel Cleeremans,et al.  Low Hopes, High Expectations , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[75]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[76]  S. Maxwell The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[77]  M. Wallach,et al.  Some Theories are Unfalsifiable , 2010 .

[78]  R. Nisbett,et al.  Culture and systems of thought: holistic versus analytic cognition. , 2001, Psychological review.

[79]  Michèle B. Nuijten,et al.  The prevalence of statistical reporting errors in psychology (1985–2013) , 2015, Behavior Research Methods.

[80]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[81]  B. Fischhoff,et al.  Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. , 1975 .

[82]  D. O. Sears College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. , 1986 .

[83]  J. L. Rogers,et al.  Using significance tests to evaluate equivalence between two experimental groups. , 1993, Psychological bulletin.

[84]  M. Branch Malignant side effects of null-hypothesis significance testing , 2014 .

[85]  B. Rossion,et al.  Fixation Patterns During Recognition of Personally Familiar and Unfamiliar Faces , 2010, Front. Psychology.

[86]  Z. Diénès Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference , 2008 .

[87]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[88]  Jay J Van Bavel,et al.  Contextual sensitivity in scientific reproducibility , 2016, Proceedings of the National Academy of Sciences.

[89]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[90]  E. Wagenmakers,et al.  Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). , 2011, Journal of personality and social psychology.

[91]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[92]  C. Stangor,et al.  Introduction to the Special Issue on Methodological Rigor and Replicability , 2016 .

[93]  William J. McGuire,et al.  A Contextualist Theory of Knowledge: Its Implications for Innovation and Reform in Psychological Research* , 1983 .

[94]  C. Gallistel,et al.  The Importance of Proving the Null , 2022 .

[95]  Gregory Francis,et al.  The Psychology of Replication and Replication in Psychology , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[96]  Andrew Gelman,et al.  Measurement error and the replication crisis , 2017, Science.

[97]  David Trafimow,et al.  The Theory of Reasoned Action , 2009 .

[98]  G. Gigerenzer,et al.  The null ritual : What you always wanted to know about significance testing but were afraid to ask , 2004 .

[99]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .