How to Improve the Quality of Peer Reviews? Three Suggestions for System-level Changes

Peer reviewing is critical in the process of legitimizing new scientific knowledge. Yet, there are concerns about its quality, especially if one considers developmental reviewing as an ideal. The present essay suggests three ways to improve review quality: provide reviewers with systematic feedback about their performance, reward active and good reviewers, and make reviewers more accountable by revealing their identities to the authors on certain conditions.

[1]  Arthur D. Santana.  Virtuous or Vitriolic , 2014 .

[2]  Munir Mandviwalla,et al.  Improving the peer review process with information technology , 2008, Decis. Support Syst..

[3]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[4]  Carol Saunders,et al.  Editor's comments: looking for diamond cutters , 2005 .

[5]  Yuko Fujigaki,et al.  Differences in knowledge production between disciplines based on analysis of paper styles and citation patterns , 1999, Scientometrics.

[6]  Gerald C. Kane,et al.  The Shoemaker's Children: Using Wikis for Information Systems Teaching, Research, and Publication , 2009, MIS Q..

[7]  J. Scott Armstrong,et al.  Research on Scientific Journals: Implications for Editors and Authors , 2005 .

[8]  Donald E. Hardaway,et al.  Open Knowledge Creation: Bringing Transparency and Inclusiveness to the Peer Review Process , 2012, MIS Q..

[9]  Richard T. Watson,et al.  How Shall We Manage Our Journals in the Future? A Discussion of Richard T. Watson's Proposals at ICIS 2004 , 2006, Commun. Assoc. Inf. Syst..

[10]  S. Rynes,et al.  The Future of Double-Blind Review in Management , 2007 .

[11]  A. Mccook.  Is Peer Review Broken , 2006 .

[12]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[13]  Juhani Iivari,et al.  DILEMMAS OF IS DEVELOPMENT , 1996 .

[14]  Detmar W. Straub,et al.  Editor's Comments: Type II Reviewing Errors and the Search for Exciting Papers , 2008 .

[15]  D. Benos,et al.  The ups and downs of peer review. , 2007, Advances in physiology education.

[16]  Martin Bichler,et al.  Design science in information systems research , 2006, Wirtschaftsinf..

[17]  David P. Lepak,et al.  Editor's Comments: What is Good Reviewing? , 2009 .

[18]  Mark A. Fine.  Reflections on enhancing accountability in the peer-review process. , 1996 .

[19]  Sandeep Purao,et al.  Action Design Research , 2011, MIS Q..

[20]  Michelle Anna Ruesch,et al.  Making the Case for Anonymity in E-Participation - An Evaluation of Real Name Policy in Gütersloh’s Second Participatory Budget , 2012 .

[21]  Carol Saunders,et al.  Editor's comments: from the trenches: thoughts on developmental reviewing , 2005 .

[22]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[23]  Kalle Lyytinen,et al.  Why the old world cannot publish? Overcoming challenges in publishing high-impact IS research , 2007, Eur. J. Inf. Syst..

[24]  Seymour Epstein,et al.  What can be done to improve the journal review process. , 1995 .

[25]  R. Blank.  The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[26]  Paul Gray,et al.  How Shall We Manage Our Journals in the Future? A Discussion of Rick Watson's ICIS 2004 Proposals , 2005, ICIS.

[27]  Les Gasser,et al.  A Design Theory for Systems That Support Emergent Knowledge Processes , 2002, MIS Q..

[28]  Mason A. Carpenter,et al.  Editor's Comments: Mentoring Colleagues in the Craft and Spirit of Peer Review , 2009 .

[29]  Juhani Iivari,et al.  Distinguishing and contrasting two strategies for design science research , 2015, Eur. J. Inf. Syst..

[30]  Glenn Regehr,et al.  To blind or not to blind? What authors and reviewers prefer , 2006, Medical education.

[31]  Allen S. Lee.  Reviewing a manuscript for publication , 1995 .