Why Replications Do Not Fix the Reproducibility Crisis: A Model and Evidence from a Large-Scale Vignette Experiment (WP-19-04)

Scientists have become increasingly concerned that “most published research findings are false” Ioannidis (2005), and have emphasized the need for replication studies. Replication entails a researcher repeating a prior research study with newly collected data. The mixed results of large-scale replication efforts have led some to conclude there is a “reproducibility crisis”: false positives are pervasive. One solution is to encourage more replications. Yet, replication studies can alter the published literature only if they actually are published. And it may well be that replication studies themselves are subject to “publication bias.” The researchers offer a micro-level model of the publication process involving an initial study and a replication. The model incorporates possible publication bias both at the initial and replication stages. This enables them to investigate the implications of publication biases on various statistical metrics of evidence quality. They then estimate the key parameters of the model with a large-scale vignette experiment conducted with political science professors teaching at Ph.D.-granting institutions in the United States. Their results show substantial evidence of publication bias: on average, respondents judged statistically significant results about 20 percentage points more likely to be published than statistically insignificant results. They further find evidence of what they call a “gotcha bias.” Replication studies that run contrary to the existing literature are more likely to be published than those consistent with past research. Publication biases at the replication stage also can lead to the appearance of increased reproducibility even when there are actually more false positive results entering the published literature. The authors thank James Dunham, Shiyao Liu, Chris Peng, Robert Pressel, Blair Read, and Jacob Rothschild for research assistance. They are grateful to Donald P. Green, Melissa Sands, and the participants at the 2017 Conference of the Society for Political Methodology and the 2018 Midwest Political Science Association Annual Meeting for useful comments and suggestions.