Modeling Evaluator Incentives when Evaluations have Consequences to the Evaluator

This paper presents a mathematical model that analyzes the rational incentives involved in providing an informative evaluation for a knowledge management system when the evaluation has some consequences for the evaluator. The analysis is carried out in the context of the peer review process that is currently used to determine which articles are published in scientific journals. The paper confirms some benefits of double blind peer review as compared to other reviewing systems. In addition, the paper uses the model to evaluate some proposed and implemented improvements to the peer review process. In conclusion, the paper outlines approaches that can result in improvements to the peer review process.

[1]  Boleslaw K. Szymanski,et al.  Selecting Scientific Papers for Publication via Citation Auctions , 2007, IEEE Intelligent Systems.

[2]  Frank Davidoff.  Improving peer review: who's responsible? , 2004, BMJ : British Medical Journal.

[3]  Stevan Harnad,et al.  Open access scientometrics and the UK Research Assessment Exercise , 2007, Scientometrics.

[4]  William Vickrey,et al.  Counterspeculation, Auctions, And Competitive Sealed Tenders , 1961 .

[5]  Robin S. Poston,et al.  Effective Use of Knowledge Management Systems: A Process Model of Content Ratings and Credibility Indicators , 2005, MIS Q..

[6]  Chang E. Koh,et al.  IS journal review process: a survey on IS research practices and journal review issues , 2003, Inf. Manag..

[7]  Assunta Arte,et al.  A Web‐based document delivery system for scientific information management in Italian research libraries , 2003 .

[8]  Detmar W. Straub,et al.  Measuring Researcher-Production in Information Systems , 2002, J. Assoc. Inf. Syst..

[9]  Eric T. Meyer.  Socio-Technical Interaction Networks: A Discussion of the Strengths, Weaknesses and Future of Kling's STIN Model , 2006, HCC.

[10]  Jérémie Gallien,et al.  Temporary and Permanent Buyout Prices in Online Auctions , 2007, Manag. Sci..

[11]  Rob Kling,et al.  Scholarly Communication and the Continuum of Electronic Publishing , 1999, J. Am. Soc. Inf. Sci..

[12]  Bradley N. Miller,et al.  MovieLens Unplugged: Experiences with a Recommender System on Four Mobile Devices , 2004 .

[13]  M. Melnik,et al.  Does a Seller's Ecommerce Reputation Matter? Evidence from Ebay Auctions , 2003 .

[14]  P. Resnick,et al.  The Market for Evaluations , 1999 .

[15]  Zoltan Nadasdy.  Electronic Journal of Cognitive and Brain Science: A Truly All-Electronic Journal: Let Democracy Replace Peer Review , 1997 .

[16]  Stefano Mizzaro,et al.  Quality control in scholarly publishing: A new proposal , 2003, J. Assoc. Inf. Sci. Technol..

[17]  Rob Kling,et al.  A BIT MORE TO IT : SCIENTIFIC MULTIPLE MEDIA COMMUNICATION FORUMS AS SOCIO-TECHNICAL INTERACTION NETWORKS , 2000 .

[18]  P D Kumar,et al.  The Future for Electronic Journals , 1996, Journal of the Royal Society of Medicine.

[19]  Steve Cayzer,et al.  Semantic blogging and decentralized knowledge management , 2004, CACM.

[20]  Morten T. Hansen,et al.  Competing for Attention in Knowledge Markets: Electronic Document Dissemination in a Management Consulting Company , 2001 .

[21]  Wanda J. Orlikowski,et al.  Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations , 2000, Theory in CSCW.

[22]  Juan Miguel Campanario,et al.  Rejecting highly cited papers: The views of scientists who encounter resistance to their discoveries from other scientists , 2007, J. Assoc. Inf. Sci. Technol..

[23]  Paul Benjamin Lowry,et al.  Global Journal Prestige and Supporting Disciplines: A Scientometric Study of Information Systems Journals , 2004, J. Assoc. Inf. Syst..

[24]  Stevan Harnad,et al.  Rational Disagreement in Peer Review , 1985 .

[25]  Roumen Vragov,et al.  Reviewing and Revamping the Double-Blind Peer Review Process , 2007 .

[26]  William H. Starbuck.  Vita Contemplativa , 2004, Duft der Zeit.

[27]  Eileen A. Hogan.  The Attention Economy: Understanding the New Currency of Business , 2001 .

[28]  Stevan Harnad,et al.  Fast-Forward on the Green Road to Open Access: The Case Against Mixing Up Green and Gold , 2005, ArXiv.