Replication in Criminology and the Social Sciences

Replication is a hallmark of science. In recent years, some medical sciences and behavioral sciences struggled with what came to be known as replication crises. As a field, criminology has yet to address formally the threats to our evidence base that might be posed by large-scale and systematic replication attempts, although it is likely we would face challenges similar to those experienced by other disciplines. In this review, we outline the basics of replication, summarize reproducibility problems found in other fields, undertake an original analysis of the amount and nature of replication studies appearing in criminology journals, and consider how criminology can begin to assess more formally the robustness of our knowledge through encouraging a culture of replication.

[1]  Matthew C. Makel,et al.  Replication of Special Education Research , 2016 .

[2]  Janet L. Lauritsen,et al.  Explaining Patterns of Offending across the Life Course: Comments on Interactional Theory and Recent Tests Based on the RYDS-RIS Data , 2005 .

[3]  M. Inzlicht,et al.  A Multilab Preregistered Replication of the Ego-Depletion Effect , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  D. Fanelli.  “Positive” Results Increase Down the Hierarchy of the Sciences , 2010, PloS one.

[5]  R. Defina,et al.  Can Incarceration Really Strip People of Racial Privilege? , 2016 .

[6]  D. Bem.  Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. , 2011, Journal of personality and social psychology.

[7]  David M. Hureau,et al.  THE TRADE IN TOOLS: THE MARKET FOR ILLICIT GUNS IN HIGH‐RISK NETWORKS , 2018, Criminology.

[8]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[9]  Jerry M. Burger,et al.  Replicating Milgram: Would people still obey today? , 2009, The American psychologist.

[10]  C. R. Tittle.  The Arrogance of Public Sociology , 2004 .

[11]  Etienne P LeBel.  A New Replication Norm for Psychology , 2015 .

[12]  Stuart J. Ritchie,et al.  Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect , 2012, PloS one.

[13]  U. Simonsohn.  Small Telescopes , 2014, Psychological science.

[14]  J. Valentine,et al.  Outcome-Reporting Bias in Education Research , 2013 .

[15]  S. McNeeley,et al.  Replication in criminology: A necessary practice , 2015 .

[16]  R. Berk,et al.  The specific deterrent effects of arrest for domestic assault. , 1984, American sociological review.

[17]  Abel Brodeur,et al.  Star Wars: The Empirics Strike Back , 2012, SSRN Electronic Journal.

[18]  U. Simonsohn.  Each Reader Decides if a Replication Counts , 2016, Psychological science.

[19]  Michael C. Frank,et al.  Response to Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[20]  W. Pridemore.  A METHODOLOGICAL ADDITION TO THE CROSS‐NATIONAL EMPIRICAL LITERATURE ON SOCIAL STRUCTURE AND HOMICIDE: A FIRST TEST OF THE POVERTY‐HOMICIDE THESIS* , 2008 .

[21]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[22]  J. Leek,et al.  What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[23]  C. Li,et al.  Problems with the γ-ray spectrum in the Fleischmann et al. experiments , 1989, Nature.

[24]  Timothy D. Wilson,et al.  Comment on “Estimating the reproducibility of psychological science” , 2016, Science.

[25]  T. P. Thornberry,et al.  Notes on Theory Construction and Theory Testing: A Response to Osgood and Lauritsen , 2005 .

[26]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[27]  S. Milgram.  BEHAVIORAL STUDY OF OBEDIENCE. , 1963, Journal of abnormal psychology.

[28]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[29]  Matthew C. Makel,et al.  Creativity is more than novelty: Reconsidering replication as a creativity act. , 2014 .

[30]  J. Ioannidis.  Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[31]  Reginald B. Adams,et al.  Investigating Variation in Replicability: A “Many Labs” Replication Project , 2014 .

[32]  Thomas C. Herndon,et al.  Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff , 2014 .

[33]  M. Fleischmann,et al.  Electrochemically Induced Nuclear Fusion of Deuterium , 1989 .

[34]  J. Ioannidis,et al.  Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. , 2016, JAMA.

[35]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[36]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[37]  John A. List,et al.  One swallow doesn’t make a summer: new evidence on anchoring effects , 2014 .

[38]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[39]  Daniele Fanelli,et al.  Negative results are disappearing from most disciplines and countries , 2011, Scientometrics.

[40]  E. Wagenmakers,et al.  Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming , 2015, Front. Psychol..

[41]  Daniel Kahneman,et al.  A new etiquette for replication. , 2014 .

[42]  R. Baumeister,et al.  Ego depletion: is the active self a limited resource? , 1998, Journal of personality and social psychology.

[43]  Matthew C. Makel,et al.  Facts Are More Important Than Novelty , 2014 .

[44]  S. Schmidt.  Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences , 2009 .

[45]  Richard Kunert.  Internal conceptual replications do not increase independent replication success , 2016, Psychonomic Bulletin & Review.

[46]  Jeffrey R. Spies,et al.  The Replication Recipe: What Makes for a Convincing Replication? , 2014 .

[47]  Matthew C. Makel,et al.  Replications in Psychology Research , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[48]  D. Borsboom,et al.  The poor availability of psychological research data for reanalysis. , 2006, The American psychologist.

[49]  Richard A. Berk,et al.  A Bayesian Analysis of the Colorado Springs Spouse Abuse Experiment , 1992 .

[50]  I. Cockburn,et al.  The Economics of Reproducibility in Preclinical Research , 2015, PLoS biology.

[51]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[52]  C. Lemons,et al.  An Inadvertent Concurrent Replication , 2016 .

[53]  Justin T. Pickett,et al.  Questionable, Objectionable or Criminal? Public Opinion on Data Fraud and Selective Reporting in Science , 2016, Science and Engineering Ethics.

[54]  J. Ioannidis.  Why Most Discovered True Associations Are Inflated , 2008, Epidemiology.

[55]  Kathryn C. Oleson,et al.  Harnessing the Undiscovered Resource of Student Research Projects , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[56]  Stanley Lieberson,et al.  B ARKING U P THE W RONG B RANCH : Scientific Alternatives to the Current Model of Sociological Science , 2002 .

[57]  John P. A. Ioannidis,et al.  Reproducible Research Practices and Transparency across the Biomedical Literature , 2016, PLoS biology.

[58]  J. Freese,et al.  Replication in Social Science , 2017 .

[59]  Michèle B. Nuijten,et al.  The prevalence of statistical reporting errors in psychology (1985–2013) , 2015, Behavior Research Methods.

[60]  D. Doliński,et al.  Would You Deliver an Electric Shock in 2015? Obedience in the Experimental Paradigm Developed by Stanley Milgram in the 50 Years Following the Original Studies , 2017 .

[61]  Susann Fiedler,et al.  Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency , 2016, PLoS biology.

[62]  Janell D. Schmidt,et al.  FROM INITIAL DETERRENCE TO LONGTERM ESCALATION: SHORT-CUSTODY ARREST FOR POVERTY GHETTO DOMESTIC VIOLENCE* , 1991 .

[63]  Robbie C. M. van Aert,et al.  Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking , 2016, Front. Psychol..

[64]  Andrew M. Penner,et al.  The Race of a Criminal Record: How Incarceration Colors Racial Perceptions , 2010 .

[65]  R. Nuzzo.  How scientists fool themselves – and how they can stop , 2015, Nature.

[66]  Yolanda Gil,et al.  Enhancing reproducibility for computational methods , 2016, Science.

[67]  Carly R. Knight,et al.  Translating Causal Claims: Principles and Strategies for Policy-Relevant Criminology , 2013 .

[68]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[69]  G. Clore,et al.  Evaluating Psychological Research Requires More Than Attention to the N , 2016, Psychological science.

[70]  R. Baumeister.  Ego Depletion and Self-Control Failure: An Energy Model of the Self's Executive Function , 2002 .

[71]  Matthew C. Makel,et al.  Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency in Research. , 2017 .

[72]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[73]  W. Pridemore.  Poverty Matters: A Reassessment of the Inequality–Homicide Relationship in Cross-National Studies , 2011 .

[74]  L. Appel,et al.  Enough Is Enough: Stop Wasting Money on Vitamin and Mineral Supplements , 2013, Annals of Internal Medicine.

[75]  R. Lall.  How Multiple Imputation Makes a Difference , 2016, Political Analysis.

[76]  Leif D. Nelson,et al.  A 21 Word Solution , 2012 .