The iSchool Community: A Case Study of iConference Reviews

A fair review process is essential to the success of any scientific conference. In this paper we present an analysis of the reviewing process of the 2014-2015 iConferences as well as a demographic analysis of the iConference community as a whole. The results show a clear need for making the reviewer pool more representative of the iSchool community as a whole by including more women and more researchers from Asian institutions. Other recommendations are to improve the continuity of the reviewer pool and to provide clearer instructions to reviewers to ensure that written reviews explicitly cover all the aspects represented by the review scores. The results of our study provide the iSchool community with a descriptive analysis of its community and a better understanding of its review process.

[1]  Amber E. Budden,et al.  To Name or Not to Name: The Effect of Changing Author Gender on Peer Review , 2009 .

[2]  Lian Pin Koh,et al.  Reviewer recommendations and editors’ decisions for a conservation journal: Is it just a crapshoot? And do Chinese authors get a fair shot? , 2015 .

[3]  Perry J Pickhardt,et al.  Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American Journal of Roentgenology. , 2005, AJR. American journal of roentgenology.

[4]  Syavash Nobarany.  Rethinking the peer review process , 2014, CSCW Companion '14.

[5]  John Dumay,et al.  Academic performance, publishing and peer review: peering into the twilight zone , 2015 .

[6]  L. Bornmann,et al.  A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants , 2010, PloS one.

[7]  Tom Tregenza,et al.  Gender bias in the refereeing process , 2002 .

[8]  Elizabeth R. Ellwood,et al.  Do gender, nationality, or academic age affect review decisions? An analysis of submissions to the journal Biological Conservation , 2009 .

[9]  H. Marsh,et al.  Improving the Peer-review Process for Grant Applications , 2022 .

[10]  C. Gross,et al.  Effect of blinded peer review on abstract acceptance. , 2006, JAMA.

[11]  Malcolm Tight.  Reviewing the reviewers , 2003 .

[12]  N. Ambady,et al.  Will a Category Cue Affect You? Category Cues, Positive Stereotypes and Reviewer Recall for Applicants , 2000 .

[13]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[14]  Michael Goldweber,et al.  Reviewing the SIGCSE reviewing process , 2008, SGCS.

[15]  S. Ceci,et al.  Understanding current causes of women's underrepresentation in science , 2011, Proceedings of the National Academy of Sciences.

[16]  Ricardo Conejo,et al.  Bias in peer review : a case study , 2018 .

[17]  Fytton Rowland,et al.  The peer‐review process , 2002, Learn. Publ..