Double-Blind Review in Software Engineering Venues: The Community's Perspective

The peer review process is central to the scientific method, the advancement and spread of research, as well as crucial for individual careers. However, the single-blind review mode currently used in most Software Engineering (SE) venues is susceptible to apparent and hidden biases, since reviewers know the identity of authors. We perform a study on the benefits and costs that are associated with introducing double-blind review in SE venues. We surveyed the SE community's opinion and interviewed experts on double-blind reviewing. Our results indicate that the costs, mostly logistic challenges and side effects, outnumber its benefits and mostly regard difficulty for authors in blinding papers, for reviewers in understanding the increment with respect to previous work from the same authors, and for organizers to manage a complex transition. While the surveyed community largely consents on the costs of DBR, only less than one-third disagree with a switch to DBR for SE journals, all SE conferences, and, in particular, ICSE, the analysis of a survey with authors of submitted papers at ICSE 2016 run by the program chairs of that edition corroborates our result.

[1]  Richard T. Snodgrass.  Frequently-asked questions about double-blind reviewing , 2007, SGMD.

[2]  E. Harmon-Jones,et al.  Neural Signals for the Detection of Unintentional Race Bias , 2004, Psychological science.

[3]  A. Furnham.  Response bias, social desirability and dissimulation , 1986 .

[4]  Bella Martin,et al.  Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions , 2012 .

[5]  Richard Walker,et al.  Emerging trends in peer review—a survey , 2015, Front. Neurosci..

[6]  R. Steinpreis,et al.  The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study , 1999 .

[7]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[8]  Kathryn S. McKinley.  More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings , 2015 .

[9]  B. Glaser.  Doing grounded theory : issues and discussions , 1998 .

[10]  J. Armstrong,et al.  Peer review for journals: Evidence on quality control, fairness, and innovation , 1997 .

[11]  H. Zeisel,et al.  Race Bias in the Administration of the Death Penalty: The Florida Experience , 1981 .

[12]  Robert P Freckleton,et al.  Does double-blind review benefit female authors? , 2008, Trends in ecology & evolution.

[13]  Shari Lawrence Pfleeger,et al.  Personal Opinion Surveys , 2008, Guide to Advanced Empirical Software Engineering.

[14]  Richard T. Snodgrass,et al.  Editorial: Single- versus double-blind reviewing , 2007, TODS.

[15]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 2 , 1998 .

[16]  Flaminio Squazzoni,et al.  Is three better than one? simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review , 2015, 2015 Winter Simulation Conference (WSC).

[17]  Flaminio Squazzoni,et al.  Does incentive provision increase the quality of peer review? An experimental study , 2013 .

[18]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[19]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 1 , 1998 .

[20]  Richard T. Snodgrass,et al.  Single- versus double-blind reviewing: an analysis of the literature , 2006, SGMD.

[21]  Howard N. Garb,et al.  Race Bias, Social Class Bias, and Gender Bias in Clinical Judgment , 1997 .

[22]  Paul J. Roebber,et al.  Peer Review, Program Officers and Science Funding , 2011, PloS one.

[23]  D. Altman,et al.  The odds ratio , 2000, BMJ : British Medical Journal.

[24]  Galen C. Britz.  Improving Performance Through Statistical Thinking , 2000 .

[25]  T. Tregenza,et al.  Double-blind review favours increased representation of female authors. , 2008, Trends in ecology & evolution.

[26]  H-U Simon,et al.  Doing Grounded Theory , 2014 .

[27]  Thomas R. Lindlof.  Qualitative Communication Research Methods , 1994 .

[28]  Lutz Bornmann,et al.  The luck of the referee draw: the effect of exchanging reviews , 2009, Learn. Publ..

[29]  Nancy Belunis,et al.  Improving Performance Through Statistical Thinking , 2001, Technometrics.

[30]  R. Blank.  The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[31]  M. El-Masri.  Odds ratio. , 2013, The Canadian nurse.

[32]  Lisa Bero,et al.  Measuring the effectiveness of scientific gatekeeping , 2014, Proceedings of the National Academy of Sciences.

[33]  Foster J. Provost,et al.  The myth of the double-blind review?: author identification using only citations , 2003, SKDD.

[34]  J. R. Gilbert,et al.  Is there gender bias in JAMA's peer review process? , 1994, JAMA.

[35]  Lee Sigelaman.  Question-Order Effects on Presidential Popularity , 1981 .

[36]  E. Harmon-Jones,et al.  The regulation of explicit and implicit race bias: the role of motivations to respond without prejudice. , 2002, Journal of personality and social psychology.

[37]  Nahid Golafshani,et al.  Understanding Reliability and Validity in Qualitative Research , 2003 .

[38]  David Lo,et al.  How practitioners perceive the relevance of software engineering research , 2015, ESEC/SIGSOFT FSE.

[39]  R. R. Snell,et al.  Menage a Quoi? Optimal Number of Peer Reviewers , 2015, PloS one.

[40]  Marcus Ciolkowski,et al.  Conducting on-line surveys in software engineering , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[41]  Joseph A. Konstan,et al.  Conference paper selectivity and impact , 2010, Commun. ACM.

[42]  Juan Miguel Campanario,et al.  Rejecting and resisting Nobel class discoveries: accounts by Nobel Laureates , 2009, Scientometrics.

[43]  Pamela P. Sawallis,et al.  Accuracy in the Identification of Scholarly and Peer-Reviewed Journals and the Peer-Review Process Across Disciplines , 2003 .

[44]  André van der Hoek,et al.  Insights and Lessons Learned from Analyzing ICSE 2014 Survey and Review Data , 2014 .

[45]  C. Goldin,et al.  Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians , 1997 .

[46]  Naomi B. Robbins,et al.  Plotting Likert and Other Rating Scales , 2011 .

[47]  Joseph Kaye,et al.  Some statistical analyses of CHI , 2009, CHI Extended Abstracts.

[48]  Michael Ley,et al.  DBLP - Some Lessons Learned , 2009, Proc. VLDB Endow..

[49]  Alberto Bacchelli,et al.  Does single blind peer review hinder newcomers? , 2017, Scientometrics.

[50]  Pradeep K. Tyagi.  The effects of appeals, anonymity, and feedback on mail survey response patterns from salespeople , 1989 .

[51]  Maureen Weicher,et al.  Peer review and secrecy in the "Information Age" , 2008, ASIST.

[52]  Mahzarin R. Banaji,et al.  Implicit Bias among Physicians and its Prediction of Thrombolysis Decisions for Black and White Patients , 2007, Journal of General Internal Medicine.

[53]  Michal Jacovi,et al.  The chasms of CSCW: a citation graph analysis of the CSCW conference , 2006, CSCW '06.

[54]  Fiona Godlee,et al.  Effect of blinding and unmasking on the quality of peer review , 1999, Journal of General Internal Medicine.

[55]  Moritz Beller,et al.  Double-blind review in software engineering venues , 2016 .