Reputation or peer review? The role of outliers

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events—papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

[1]  Dianne Millen,et al.  Some Methodological and Epistemological Issues Raised by Doing Feminist Research on Non-Feminist Women , 1997 .

[2]  L. Trinquart,et al.  The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise , 2016, PloS one.

[3]  Stephen J. Cowley,et al.  How peer-review constrains cognition: on the frontline in the knowledge sector , 2015, Front. Psychol..

[4]  B. Smoller.  Systems review. , 2010, Maryland medicine : MM : a publication of MEDCHI, the Maryland State Medical Society.

[5]  Matthias Meyer,et al.  Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models of Science? , 2011, J. Artif. Soc. Soc. Simul..

[6]  C. Lortie.  Money for nothing and your referees for free , 2011 .

[7]  N. Gilbert.  A Simulation of the Structure of Academic Science , 1997 .

[8]  Jordi Sabater-Mir,et al.  Computational trust and reputation models for open multi-agent systems: a review , 2013, Artificial Intelligence Review.

[9]  Mario Paolucci,et al.  Mechanism change in a simulation of peer review: from junk support to elitism , 2014, Scientometrics.

[10]  Robert L. Goldstone,et al.  The simultaneous evolution of author and paper networks , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Jordi Sabater-Mir,et al.  Simulating Research Behaviour , 2011, MABS.

[12]  Nicolas Payette,et al.  Agent-Based Models of Science , 2012 .

[13]  Simone Righi,et al.  The miracle of peer review and development in science: an agent-based model , 2016, Scientometrics.

[14]  R. Perrucci,et al.  From Little Science to Big Science , 2017 .

[15]  Ron Sun,et al.  Cognitive simulation of academic science , 2009, 2009 International Joint Conference on Neural Networks.

[16]  Ingo Scholtes,et al.  Quantifying the effect of editor–author relations on manuscript handling times , 2017, Scientometrics.

[17]  Flaminio Squazzoni,et al.  Do Editors Have A Silver Bullet? An Agent-Based Model Of Peer Review , 2014, ECMS.

[18]  Lutz Bornmann,et al.  Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references , 2014, J. Assoc. Inf. Sci. Technol..

[19]  Philippe Ravaud,et al.  Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data , 2015, Scientometrics.

[20]  Nicole J. Saam,et al.  Lotka's law reconsidered: The evolution of publication and citation distributions in scientific fields , 1999, Scientometrics.

[21]  Stefano Allesina,et al.  Modeling peer review: an agent-based approach , 2012 .

[22]  Santo Fortunato,et al.  Attention Decay in Science , 2015, J. Informetrics.

[23]  Philippe Ravaud,et al.  Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication , 2017, Scientometrics.

[24]  Claudio Gandelli,et al.  Opening the Black-Box of Peer Review: An Agent-Based Model of Scientist Behaviour , 2013, J. Artif. Soc. Soc. Simul..

[25]  Bruce Edmonds,et al.  Simulating the Social Processes of Science , 2011, J. Artif. Soc. Soc. Simul..

[26]  César García-Díaz,et al.  Exploring Transitions Towards Sustainable Construction: The Case of Near-Zero Energy Buildings in the Netherlands , 2015, J. Artif. Soc. Soc. Simul..

[27]  David Cornforth,et al.  Modelling Academics as Agents: An Implementation of an Agent-Based Strategic Publication Model , 2015, J. Artif. Soc. Soc. Simul..

[28]  M. Biagioli.  From Book Censorship to Academic Peer Review , 2002 .

[29]  Myong Kee Jeong,et al.  Two-phase edge outlier detection method for technology opportunity discovery , 2017, Scientometrics.

[30]  Peter Frishauf,et al.  Reputation Systems: A New Vision for Publishing and Peer Review , 2009 .

[31]  Sergey Popov,et al.  Alma mat(t)er(s): Determinants of early career success in economics , 2014, PloS one.

[32]  Chrysanthos Dellarocas.  Designing Reputation Systems for the Social Web , 2010 .

[33]  Mario Paolucci,et al.  A Simulation Of Disagreement For Control Of Rational Cheating In Peer Review , 2012, ECMS.