Replication in Social Science

Across the medical and social sciences, new discussions about replication have led to transformations in research practice. Sociologists, however, have been largely absent from these discussions. The goals of this review are to introduce sociologists to these developments, synthesize insights from science studies about replication in general, and detail the specific issues regarding replication that occur in sociology. The first half of the article argues that a sociologically sophisticated understanding of replication must address both the ways that replication rules and conventions evolved within an epistemic culture and how those cultures are shaped by specific research challenges. The second half outlines the four main dimensions of replicability in quantitative sociology—verifiability, robustness, repeatability, and generalizability—and discusses the specific ambiguities of interpretation that can arise in each. We conclude by advocating some commonsense changes to promote replication while acknowled...

[1]  R. Mack.  The Need for Replication Research in Sociology , 1951 .

[2]  Generalizations in the Social Sciences , 1953, American Journal of Sociology.

[3]  F. D. Wilson,et al.  The Replication Problem in Sociology: A Report and a Suggestion* , 1973 .

[4]  H. M. Collins,et al.  The TEA Set: Tacit Knowledge and Scientific Networks , 1974 .

[5]  T. Pinch.  Normal Explanations of the Paranormal: The Demarcation Problem and Fraud in Parapsychology , 1979 .

[6]  G.D.L. Travis.  Replicating Replication? Aspects of the Social Construction of Learning in Planarian Worms , 1981 .

[7]  Theodore Caplow,et al.  MIDDLETOWN III: PROBLEMS OF REPLICATION, LONGITUDINAL MEASUREMENT, AND TRIANGULATION , 1983 .

[8]  M. Mulkay.  The Scientist Talks Back: A One-Act Play, with a Moral, about Replication in Science and Reflexivity in Sociology , 1984 .

[9]  Henry A. Walker,et al.  Scope Statements: Imperatives for Evaluating Theory , 1985 .

[10]  Harold Maurice Collins,et al.  The meaning of replication and the science of economics , 1991 .

[11]  Lorraine Daston,et al.  Objectivity and the Escape from Perspective , 1992 .

[12]  Richard R. Peterson.  A re-evaluation of the economic consequences of divorce. , 1996 .

[13]  Statistical errors, faulty conclusions, misguided policy : Reply to Weitzman , 1996 .

[14]  Martyn Hammersley,et al.  Qualitative Data Archiving: Some Reflections on its Prospects and Problems , 1997 .

[15]  H. Collins.  The Meaning of Data: Open and Closed Evidential Cultures in the Search for Gravitational Waves1 , 1998, American Journal of Sociology.

[16]  R. Baumeister,et al.  Ego depletion: is the active self a limited resource? , 1998, Journal of personality and social psychology.

[17]  N. Kerr.  HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[18]  N. Cartwright.  The dappled world : a study of the boundaries of science , 1999 .

[19]  Daniel Kennefick,et al.  Star Crushing: , 2000 .

[20]  S. Kanazawa,et al.  On Roland K. Hawkes's "Spatial Patterning of Urban Population Characteristics" , 1975, American Journal of Sociology.

[21]  Brian Powell,et al.  Making Love out of Nothing at All? Null Findings and the Trivers‐Willard Hypothesis1 , 2001, American Journal of Sociology.

[22]  Harold Maurice Collins,et al.  Tacit Knowledge, Trust and the Q of Sapphire , 2001 .

[23]  H. M. Collins,et al.  The Third Wave of Science Studies , 2002, Science, Technology, and Society.

[24]  Robert J. Barro,et al.  Religion and Economic Growth across Countries , 2003, American Sociological Review.

[25]  J. Freese,et al.  Toward Some Fundamentals of Fundamental Causality: Socioeconomic Status and Health in the Routine Clinic Visit for Diabetes1 , 2005, American Journal of Sociology.

[26]  D. Borsboom,et al.  The poor availability of psychological research data for reanalysis. , 2006, The American psychologist.

[27]  Matthew E. Brashears,et al.  Social Isolation in America: Changes in Core Discussion Networks over Two Decades , 2006 .

[28]  Teresa D. Harrison,et al.  Lessons from the JMCB Archive , 2006 .

[29]  Jeff Manza,et al.  Social Policy Responsiveness in Developed Democracies , 2006 .

[31]  R. Putnam,et al.  E Pluribus Unum: Diversity and Community in the Twenty‐first Century The 2006 Johan Skytte Prize Lecture , 2007 .

[32]  E. Leahey.  Overseeing Research Practice , 2008 .

[33]  Cristobal Young,et al.  Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth , 2009 .

[34]  M. Small.  `How many cases do I need?' , 2009 .

[35]  Kenneth S. Rogoff,et al.  Growth in a Time of Debt , 2010 .

[36]  Michelle J. Budig,et al.  Differences in Disadvantage , 2010 .

[37]  A. Moravcsik,et al.  Active Citation: A Precondition for Replicable Qualitative Research , 2010, PS: Political Science & Politics.

[38]  D. Bem.  Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. , 2011, Journal of personality and social psychology.

[39]  Christine L Borgman,et al.  Science friction: Data, metadata, and collaboration , 2011, Social studies of science.

[40]  Etienne P LeBel,et al.  Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice , 2011, Review of General Psychology.

[41]  Han L. J. van der Maas,et al.  Science Perspectives on Psychological an Agenda for Purely Confirmatory Research on Behalf Of: Association for Psychological Science , 2022 .

[42]  Emma Frow,et al.  Drawing a line: Setting guidelines for digital image processing in scientific journal articles , 2012 .

[43]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[44]  Stuart J. Ritchie,et al.  Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect , 2012, PloS one.

[45]  Richard Biernacki.  Reinventing evidence in social inquiry : decoding facts and variables , 2012 .

[46]  Marek N. Posard,et al.  Considerations on the ‘Replication Problem’ in Sociology , 2013 .

[47]  J. Bearak,et al.  Is the Motherhood Penalty Larger for Low-Wage Women? A Comment on Quantile Regression , 2014 .

[48]  Michelle J. Budig,et al.  Statistical Models and Empirical Evidence for Differences in the Motherhood Penalty across the Earnings Distribution , 2014 .

[49]  D. Green,et al.  When contact changes minds: An experiment on transmission of support for gay equality , 2014, Science.

[50]  M. Brent Donnellan,et al.  Commentary and Rejoinder on Johnson, Cheung, and Donnellan (2014a) , 2014 .

[51]  E. Eich.  Business Not as Usual , 2014, Psychological science.

[52]  Thomas C. Herndon,et al.  Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff , 2014 .

[53]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[54]  Alex O Holcombe,et al.  An Introduction to Registered Replication Reports at Perspectives on Psychological Science , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[55]  Daniel Kahneman,et al.  A new etiquette for replication. , 2014 .

[56]  Jeffrey R. Spies,et al.  The Replication Recipe: What Makes for a Convincing Replication? , 2014 .

[57]  Annette N. Brown,et al.  Quality evidence for policymaking: I’ll believe it when I see the replication , 2014 .

[58]  Nate Breznau.  The Missing Main Effect of Welfare State Regimes: A Replication of 'Social Policy Responsiveness in Developed Democracies' by Brooks and Manza , 2015 .

[59]  Statins for people at low risk , 2015, BMJ : British Medical Journal.

[60]  C. Abramson,et al.  Beyond Text , 2015, Sociological methodology.

[61]  David Peterson,et al.  All That Is Solid , 2015 .

[62]  Paul W. Eastwick,et al.  Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. , 2015, Journal of personality and social psychology.

[63]  J. Isaac.  For a More Public Political Science , 2015, Perspectives on Politics.

[64]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[65]  Delia Baldassarri,et al.  Love Thy Neighbor? Ethnoracial Diversity and Trust Reexamined1 , 2015, American Journal of Sociology.

[66]  Andrew V. Papachristos,et al.  Trust Thy Crooked Neighbor , 2016 .

[67]  Timothy D. Wilson,et al.  Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[68]  K. Vohs,et al.  Misguided Effort With Elusive Implications , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[69]  Alexandra K. Murphy,et al.  Ethnographic Masking in an Era of Data Transparency , 2016 .

[70]  The Emergence of Forensic Objectivity , 2016 .

[71]  R. Lall.  How Multiple Imputation Makes a Difference , 2016, Political Analysis.

[72]  M. Inzlicht,et al.  A Multilab Preregistered Replication of the Ego-Depletion Effect , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[73]  Michael C. Frank,et al.  Response to Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[74]  Harold Maurice Collins.  Reproducibility of experiments: the experimenter's regress, statistical uncertainty principle and the replication imperative , 2016 .

[75]  Michael A. Clemens,et al.  The Meaning of Failed Replications: A Review and Proposal , 2015, Social Science Research Network.