Use of politeness strategies in signed open peer review

Scholarly peer review is a complex collaborative activity that is increasingly supported by web‐based systems, yet little is known about how reviewers and authors interact in such environments, how criticisms are conveyed, or how the systems may affect the interactions and use of language of reviewers and authors. We looked at one aspect of the interactions between reviewers and authors, the use of politeness in reviewers' comments. Drawing on Brown and Levinson's (1987) politeness theory, we analyzed how politeness strategies were employed by reviewers to mitigate their criticisms in an open peer‐review process of a special track of a human‐computer interaction conference. We found evidence of frequent use of politeness strategies and that open peer‐review processes hold unique challenges and opportunities for using politeness strategies. Our findings revealed that less experienced researchers tended to express unmitigated criticism more often than did experienced researchers, and that reviewers tended to use more positive politeness strategies (e.g., compliments) toward less experienced authors. Based on our findings, we discuss implications for research communities and the design of peer‐reviewing processes and the information systems that support them.

[1]  Fiona Godlee,et al.  Effect of blinding and unmasking on the quality of peer review , 1999, Journal of General Internal Medicine.

[2]  Joanne McGrath Cohoon,et al.  Gender and computing conference papers , 2011, Commun. ACM.

[3]  Greg Myers Politeness and Certainty: The Language of Collaboration in an Al Project , 1991 .

[4]  C. Gross,et al.  Effect of blinded peer review on abstract acceptance. , 2006, JAMA.

[5]  R. Lakoff,et al.  Talking power : the politics of language in our lives , 1990 .

[6]  Bernd Pulverer,et al.  A transparent black box , 2010, The EMBO journal.

[7]  Sage L. Graham,et al.  Disagreeing to agree: Conflict, (im)politeness and identity in a computer-mediated community , 2007 .

[8]  M. Kourilova,et al.  COMMUNICATIVE CHARACTERISTICS OF REVIEWS OF SCIENTIFIC PAPERS WRITTEN BY NON-NATIVE USERS OF ENGLISH , 1998 .

[9]  E. Goffman On face-work; an analysis of ritual elements in social interaction. , 1955, Psychiatry.

[10]  Ruth Collins-Nakai The Canadian Medical Association Journal. , 2006, The New England journal of medicine.

[11]  Donna M. Johnson Compliments and Politeness in Peer-review Texts , 1992 .

[12]  S. Ceci,et al.  Peer-review practices of psychological journals: The fate of published articles, submitted again , 1982, Behavioral and Brain Sciences.

[13]  Jo Mackiewicz,et al.  The Technical Editor as Diplomat: Linguistic Strategies for Balancing Clarity and Politeness , 2003 .

[14]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[15]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 1 , 1998 .

[16]  F. Al-Hindawi,et al.  A Critique of Politeness Theories , 2016 .

[17]  Molly C Dougherty,et al.  Blinding in peer review: the preferences of reviewers for nursing journals. , 2008, Journal of advanced nursing.

[18]  Ken Hyland,et al.  Sugaring the Pill: Praise and Criticism in Written Feedback. , 2001 .

[19]  R. Herbert,et al.  The Ethnography of English Compliments and Compliment Responses : a Contrastive Sketch , 1989 .

[20]  井出 祥子,et al.  Broadening the horizon of linguistic politeness , 2005 .

[21]  R. Melero,et al.  Referees’ Attitudes toward Open Peer Review and Electronic Transmission of Papers , 2001 .

[22]  P. Hobbs,et al.  The medium is the message: politeness strategies in men's and women's voice mail messages , 2003 .

[23]  Pearl Brereton,et al.  Presenting software engineering results using structured abstracts: a randomised experiment , 2008, Empirical Software Engineering.

[24]  T R Einarson,et al.  Quality of nonstructured and structured abstracts of original research articles in the British Medical Journal, the Canadian Medical Association Journal and the Journal of the American Medical Association. , 1994, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[25]  Maryelis Pabón Berbesí,et al.  Collegiality, critique and the construction of scientific argumentation in medical book reviews: A diachronic approach , 2007 .

[26]  George Lakoff,et al.  Hedges: A study in meaning criteria and the logic of fuzzy concepts , 1973, J. Philos. Log..

[27]  Barbara M. Wildemuth,et al.  Applications of Social Research Methods to Questions in Information and Library Science , 2009 .

[28]  Lawrence Souder,et al.  The ethics of scholarly peer review: a review of the literature , 2011, Learn. Publ..

[29]  J. McGrath Cohoon,et al.  Gender and Computing , 2015 .

[30]  Greg Myers,et al.  The pragmatics of politeness in scientific articles , 1989 .

[31]  Kevin Crowston,et al.  Goals and Tasks: Two Typologies of Citizen Science Projects , 2012, 2012 45th Hawaii International Conference on System Sciences.

[32]  E. Ernst,et al.  Reviewer bias: a blinded experimental study. , 1994, The Journal of laboratory and clinical medicine.

[33]  M. Mahoney Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[34]  Paula Maier,et al.  Politeness Strategies in Business Letters by Native and Non-Native English Speakers. , 1992 .

[35]  Lutz Bornmann,et al.  Scientific peer review , 2011, Annu. Rev. Inf. Sci. Technol..

[36]  D. Shatz Peer Review: A Critical Inquiry , 2004 .

[37]  Richard J. Lipton,et al.  Social processes and proofs of theorems and programs , 1977, POPL.

[38]  D. Benos,et al.  The ups and downs of peer review. , 2007, Advances in physiology education.

[39]  Terrence A. Brooks,et al.  Evidence of complex citer motivations , 1986, J. Am. Soc. Inf. Sci..

[40]  J E Riggs,et al.  Priority, Rivalry, and Peer Review , 1995, Journal of child neurology.

[41]  Manuel Núñez-Nickel,et al.  The multilayered nature of reference selection , 2009, J. Assoc. Inf. Sci. Technol..

[42]  R. Blank The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[43]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[44]  R Smith,et al.  Opening up BMJ peer review , 1999, BMJ.

[45]  R. Fletcher,et al.  The effects of blinding on the quality of peer review. A randomized trial. , 1990, JAMA.

[46]  Axel Hübler,et al.  Understatements and Hedges in English , 1983 .

[47]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[48]  Norman Kaplan,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1974 .

[49]  T. Tyler,et al.  The Social Psychology of Procedural Justice , 1988 .

[50]  Stephen I. Abramowitz,et al.  Publish or Politic: Referee Bias in Manuscript Review1 , 1975 .

[51]  Erina L. MacGeorge,et al.  The Evaluation of Advice in Supportive Interactions: Facework and Contextual Factors. , 2002 .

[52]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[53]  Manuel Núñez-Nickel,et al.  The multilayered nature of reference selection , 2009 .

[54]  J. Armstrong,et al.  Peer review for journals: Evidence on quality control, fairness, and innovation , 1997 .

[55]  D. Rennie,et al.  Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. , 1998, JAMA.

[56]  T. Holtgraves,et al.  Interpersonal underpinnings of request strategies: general principles and differences due to culture and gender. , 1992, Journal of personality and social psychology.

[57]  N. Black,et al.  Effect of blinding and unmasking on the quality of peer review: a randomized trial. , 1998, JAMA.

[58]  Lutz Bornmann,et al.  Reliability of reviewers' ratings when using public peer review: a case study , 2010, Learn. Publ..

[59]  Dale T. Miller,et al.  Disrespect and the experience of injustice. , 2001, Annual review of psychology.

[60]  R. Watts,et al.  Relevance and relational work: linguistic politeness as politic behavior , 1989 .

[61]  Louise Hall,et al.  Peer review in a changing world: An international study measuring the attitudes of researchers , 2013, J. Assoc. Inf. Sci. Technol..

[62]  Joan Cutting,et al.  Vague Language Explored , 2007 .

[63]  T. Tregenza,et al.  Double-blind review favours increased representation of female authors. , 2008, Trends in ecology & evolution.

[64]  Deborah Tannen,et al.  Talking from 9 to 5 : women and men in the workplace : language, sex and power , 1995 .

[65]  D. Chubin,et al.  Peerless Science: Peer Review and U. S. Science Policy , 1990 .

[66]  Laura Callahan Asking for a Letter of Recommendation in Spanish and English: A Pilot Study of Face Strategies , 2011, Hispania.

[67]  K. Tracy,et al.  “Reasonable Hostility”: Situation-appropriate face-attack , 2008 .

[68]  Nigel Harwood An interview-based study of the functions of citations in academic writing across two disciplines , 2009 .

[69]  Catherine Kerbrat-Orecchioni,et al.  A Multilevel Approach in the Study of Talk-In-Interaction , 1997 .

[70]  Robert E. Kraut,et al.  Mind your Ps and Qs: the impact of politeness and rudeness in online communities , 2008, CSCW.

[71]  Janet Holmes,et al.  Paying compliments: A sex-preferential politeness strategy , 1988 .

[72]  Inmaculada Fortanet,et al.  Evaluative language in peer review referee reports , 2008 .

[73]  Karim Khan Is open peer review the fairest system? No , 2010, BMJ : British Medical Journal.

[74]  Marc N. Elliott,et al.  Using Pooled Kappa to Summarize Interrater Agreement across Many Items , 2008 .

[75]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[76]  R. Sternberg On Civility in Reviewing , 2002 .

[77]  J. Scott Peer Review for Journals: Evidence on Quality Control, Fairness, and Innovation , 1997 .

[78]  E. Goffman,et al.  The Territories of the Self , 2017 .

[79]  S. Isenberg,et al.  The effect of masking manuscripts for the peer-review process of an ophthalmic journal , 2009, British Journal of Ophthalmology.

[80]  Yan Zhang,et al.  Qualitative Analysis of Content by , 2005 .

[81]  G. Wilkinson,et al.  Open peer review: A randomised controlled trial , 2000, British Journal of Psychiatry.

[82]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[83]  M. Hogg,et al.  Attitudes, behavior, and social context: The role of norms and group membership in social influence processes , 2000 .

[84]  Tony Delamothe,et al.  Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial , 2010, BMJ : British Medical Journal.

[85]  C. Bingham,et al.  The Medical Journal of Australia internet peer-review study , 1998, The Lancet.

[86]  Cassidy R. Sugimoto,et al.  Citation gamesmanship: testing for evidence of ego bias in peer review , 2012, Scientometrics.

[87]  Erving Goffman,et al.  The Nature of Deference and Demeanor , 1956 .

[88]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations.@@@Einstein and the Generations of Science. , 1976 .

[89]  P J Bieling,et al.  When reviews attack: ethics, free speech, and the peer review process. , 2000, Canadian psychology = Psychologie canadienne.

[90]  S. Gilliland,et al.  PROCEDURAL AND DISTRIBUTIVE JUSTICE IN THE EDITORIAL REVIEW PROCESS , 1996 .

[91]  Lutz Bornmann,et al.  What do citation counts measure? A review of studies on citing behavior , 2008, J. Documentation.

[92]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 2 , 1998 .

[93]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.