Standard Operating Procedures: A Safety Net for Pre-Analysis Plans

ABSTRACT Across the social sciences, growing concerns about research transparency have led to calls for pre-analysis plans (PAPs) that specify in advance how researchers intend to analyze the data they are about to gather. PAPs promote transparency and credibility by helping readers distinguish between exploratory and confirmatory analyses. However, PAPs are time-consuming to write and may fail to anticipate contingencies that arise in the course of data collection. This article proposes the use of “standard operating procedures” (SOPs)—default practices to guide decisions when issues arise that were not anticipated in the PAP. We offer an example of an SOP that can be adapted by other researchers seeking a safety net to support their PAPs.

[1]  Neil Malhotra,et al.  Underreporting in Psychology Experiments , 2016 .

[2]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[3]  B. Nyhan.  Increasing the Credibility of Political Science Research: A Proposal for Journal Reforms , 2015, PS: Political Science & Politics.

[4]  Benjamin A. Olken,et al.  Promises and Perils of Pre-analysis Plans , 2015 .

[5]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[6]  James E. Monogan.  Research Preregistration in Political Science: The Case, Counterarguments, and a Response to Critiques , 2015, PS: Political Science & Politics.

[7]  Debates: The Impact of Voter Knowlegde Initiaties in Sierra Leone , 2015 .

[8]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[9]  S. Muthukumaraswamy,et al.  Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond , 2014 .

[10]  Brian A. Nosek,et al.  Promoting Transparency in Social Science Research , 2014, Science.

[11]  Kelly Bidwella,et al.  The Impact of Voter Knowledge Initiatives in Sierra Leone , 2014 .

[12]  David Moher,et al.  SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials , 2013, BMJ.

[13]  Macartan Humphreys,et al.  Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration , 2012, Political Analysis.

[14]  James E. Monogan.  A Case for Registering Studies of Political Outcomes: An Application in the 2010 House Elections , 2013, Political Analysis.

[15]  Abel Brodeur,et al.  Star Wars: The Empirics Strike Back , 2012, SSRN Electronic Journal.

[16]  Christian P. Robert,et al.  Large-scale inference , 2010 .

[17]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[18]  Edward Miguel,et al.  Reshaping Institutions: Evidence on Aid Impacts Using a Pre-Analysis Plan , 2011 .

[19]  J. Felson,et al.  Statistical Models and Causal Inference: A Dialogue with the Social Sciences , 2011 .

[20]  P. Westfall,et al.  Multiple comparisons and multiple tests using SAS , 2011 .

[21]  Philip B. Stark,et al.  Statistical Models and Causal Inference: A Dialogue with the Social Sciences , 2009 .

[22]  Philip B. Stark,et al.  Statistical Models and Causal Inference: Survival Analysis: An Epidemiological Hazard? , 2009 .

[23]  Michael L. Anderson.  Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects , 2008 .

[24]  Alan S. Gerber,et al.  Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals , 2008 .

[25]  David A. Freedman,et al.  Editorial: Oasis or Mirage? , 2008 .

[26]  D. Rubin.  The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials , 2007, Statistics in medicine.

[27]  Matthew Rabin,et al.  Choice and Procrastination , 2000 .

[28]  J. Tukey.  Tightening the clinical trial. , 1993, Controlled clinical trials.

[29]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[30]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[31]  David A Freedman.  Oasis or Mirage ? , 2022 .