Ranking Games

Background: Research rankings based on bibliometrics today dominate governance in academia and determine careers in universities. Method: Analytical approach to capture the incentives by users of rankings and by suppliers of rankings, both on an individual and an aggregate level. Result: Rankings may produce unintended negative side effects. In particular, rankings substitute the “taste for science” by a “taste for publication.” We show that the usefulness of rankings rests on several important assumptions challenged by recent research. Conclusion: We suggest as alternatives careful socialization and selection of scholars, supplemented by periodic self-evaluations and awards. The aim is to encourage controversial discourses in order to contribute meaningful to the advancement of science.

[1]  C. Chow,et al.  Are Articles in “Top” Management Journals Necessarily of Higher Quality? , 2007 .

[2]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .

[3]  E. Deci,et al.  Self‐determination theory and work motivation , 2005 .

[4]  Roni Reiter-Palmon,et al.  Encyclopedia of Creativity , 2011 .

[5]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[6]  Edwin A. Locke,et al.  Has Goal Setting Gone Wild, or Have Its Attackers Abandoned Good Scholarship? , 2009 .

[7]  Michael Power,et al.  The Theory of the Audit Explosion , 2007 .

[8]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[9]  Nina Mazar,et al.  Large stakes and big mistakes , 2009 .

[10]  Philip Campbell,et al.  Escape from the impact factor , 2008 .

[11]  J. Hudson.  Ranking Journals , 2013 .

[12]  Teresa M. Amabile,et al.  Assessing the Work Environment for Creativity , 1996 .

[13]  P. Seglen,et al.  Education and debate , 1999, The Ethics of Public Health.

[14]  A. Sokal,et al.  Revelation: A Physicist Experiments with Cultural Studies , 2000 .

[15]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[16]  Randall G. Holcombe,et al.  The National Research Council Ranking of Research Universities: Its Impact on Research in Economics , 2004 .

[17]  Bruno S. Frey,et al.  Publishing as Prostitution? – Choosing Between One's Own Ideas and Academic Success , 2003 .

[18]  P. David,et al.  Toward a new economics of science , 1994 .

[19]  J. Hicks,et al.  The economics of science , 1996 .

[20]  E. Swanson,et al.  Publishing in the Majors: A Comparison of Accounting, Finance, Management, and Marketing , 2002 .

[21]  Frederic S. Lee.  The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities , 2006 .

[22]  Richard Smith,et al.  Journal accused of manipulating impact factor , 1997 .

[23]  Bruno S. Frey,et al.  Economists in the PITS? , 2009, SSRN Electronic Journal.

[24]  David Cyranoski,et al.  Cash for papers: putting a premium on publication , 2006, Nature.

[25]  Bruno S. Frey,et al.  Are more and better indicators the solution , 2009 .

[26]  Andrew J. Oswald,et al.  How Should Peer�?Review Panels Behave? , 2013 .

[27]  R. Monastersky.  The number that's devouring science , 2005 .

[28]  B. Clark.  Creating entrepreneurial universities : organizational pathways of transformation , 1998 .

[29]  D. Laband.  On the Use and Abuse of Economics Journal Rankings , 2013 .

[30]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[31]  Margit Osterloh,et al.  Governance by Numbers. Does It Really Work in Research? , 2010 .

[32]  Martin Emmert,et al.  Effects of pay for performance in health care: a systematic review of systematic reviews. , 2013, Health policy.

[33]  Susanne Neckermann,et al.  Awards: A View from Psychological Economics , 2008 .

[34]  H. Browman,et al.  Factors and indices are one thing, deciding who is scholarly, why they are scholarly, and the relative value of their scholarship is something else entirely , 2008 .

[35]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[36]  J. Gans,et al.  How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists , 1994 .

[37]  Bruno S. Frey,et al.  Awards are a Special Kind of Signal , 2014 .

[38]  Hugh Willmott,et al.  Just How Managed is the McUniversity? Craig Prichard, Hugh Willmott , 1997 .

[39]  B. Frey,et al.  Does the John Bates Clark Medal Boost Subsequent Productivity and Citation Success? , 2013, SSRN Electronic Journal.

[40]  Henk F. Moed,et al.  The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review , 2007 .

[41]  Peter Taylor,et al.  Citation Statistics , 2009, ArXiv.

[42]  William H. Starbuck,et al.  The production of knowledge : the challenge of social science research , 2006 .

[43]  Dan L. Worrell,et al.  Assessing Business Scholarship: The Difficulties in Moving Beyond the Rigor-Relevance Paradigm Trap , 2009 .

[44]  Lutz Bornmann,et al.  The luck of the referee draw: the effect of exchanging reviews , 2009, Learn. Publ..

[45]  B. Frey,et al.  Organizational Control Systems and Pay-for-Performance in the Public Service , 2013 .

[46]  J. Evans,et al.  Quotational and reference accuracy in surgical journals. A continuing peer review problem. , 1990, JAMA.

[47]  S. Albers.  Misleading Rankings of Research in Business , 2009 .

[48]  D. Horrobin,et al.  Peer review of grant applications: a harbinger for mediocrity in clinical research? , 1996, The Lancet.

[49]  Symonds Em.  Research assessment exercise , 1998 .

[50]  L. Butler,et al.  Assessing university research: A plea for a balanced approach , 2007 .

[51]  B. Frey,et al.  Motivation crowding theory , 2001 .

[52]  A. Patton,et al.  Professions , 1934, Science.

[53]  Margit Osterloh,et al.  Governing Knowledge Work:: Transactional and Transformational Solutions , 2010 .

[54]  D. Simonton.  Creativity in Science: Chance, Logic, Genius, and Zeitgeist , 2004 .

[55]  Donald Gillies,et al.  Hempelian and Kuhnian approaches in the philosophy of medicine: the Semmelweis case. , 2005, Studies in history and philosophy of biological and biomedical sciences.

[56]  Eric W. K. Tsang,et al.  The As-Is Journal Review Process: Let Authors Own Their Ideas , 2006 .

[57]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[58]  Robert D. Tollison,et al.  Dry Holes in Economic Research , 2003 .

[59]  Heinz Maier-Leibnitz.  The measurement of quality and reputation in the world of learning , 1989 .

[60]  W. Ouchi.  A Conceptual Framework for the Design of Organizational Control Mechanisms , 1979 .

[61]  J. Kay.  Obliquity , 2013 .

[62]  Richard Whitley,et al.  Changing Governance and Authority Relations in the Public Sciences , 2011 .

[63]  N. Adler,et al.  When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings , 2009 .

[64]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 1 , 1998 .

[65]  Arthur G. Bedeian,et al.  Peer Review and the Social Construction of Knowledge in the Management Discipline , 2004 .

[66]  Janne S. Kotiaho,et al.  Unfamiliar citations breed mistakes , 1999, Nature.

[67]  Maxwell L. King,et al.  Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: blunt instruments or inspired metrics? , 2009 .

[68]  E. Deci,et al.  A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. , 1999, Psychological bulletin.

[69]  A. Oswald.  An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers , 2006, SSRN Electronic Journal.

[70]  R. Eisenberger,et al.  Detrimental effects of reward. Reality or myth? , 1996, The American psychologist.

[71]  M. Polanyi.  The Republic of Science: Its Political and Economic Theory , 1962 .

[72]  Wendy Nelson Espeland,et al.  The Discipline of Rankings: Tight Coupling and Organizational Change , 2009 .

[73]  W. Espeland,et al.  Rankings and Reactivity: How Public Measures Recreate Social Worlds1 , 2007, American Journal of Sociology.

[74]  Alfred Kieser,et al.  How much do specialists have to learn from each other when they jointly develop radical product innovations , 2008 .

[75]  Bambi M. Douma,et al.  Goal Setting as a Motivator of Unethical Behavior , 2004 .

[76]  J. V. Heilig,et al.  Accountability Texas-Style: The Progress and Learning of Urban Minority Students in a High-Stakes Testing Context , 2008 .

[77]  Philip Mirowski,et al.  Science Bought and Sold: Essays in the Economics of Science , 2002 .

[78]  N J Hansen,et al.  Professionalism , 2017, Getting Ready for Direct Practice in Social Work.

[79]  J. Hahn.  Victims Of Groupthink A Psychological Study Of Foreign Policy Decisions And Fiascoes , 2016 .

[80]  Bruno S. Frey,et al.  Motivate people with prizes , 2010 .

[81]  K. Arrow.  Economic Welfare and the Allocation of Resources for Invention , 1962 .

[82]  D. Laitin.  Marginality , 1995 .

[83]  T. Porter,et al.  Trust in numbers: The pursuit of objectivity in science and public life , 1996 .

[84]  H. Ursprung,et al.  Who is the ”Platz-Hirsch“ of the German Economics Profession? , 2007 .

[85]  B. Stensaker,et al.  Doomed to be Entrepreneurial: Institutional Transformation or Institutional Lock-Ins of ‘New’ Universities? , 2013 .

[86]  E. Ferlie,et al.  The Oxford Handbook of Public Management , 2007 .

[87]  Paula E. Stephan.  Science and the University: Challenges for Future Research , 2008 .

[88]  Nikolaus Kriegeskorte,et al.  Open Evaluation: A Vision for Entirely Transparent Post-Publication Peer Review and Rating for Science , 2012, Front. Comput. Neurosci..

[89]  Russel E. Kaufman,et al.  “Science, the Endless Frontier” , 1960, Nature.

[90]  D. Campbell.  Factors relevant to the validity of experiments in social settings. , 1957, Psychological bulletin.

[91]  Franck Laloë,et al.  Bibliometric evaluation of individual researchers: not even right... not even wrong! , 2009 .

[92]  Siegwart Lindenberg.  Intrinsic motivation in a new light , 2001 .

[93]  Pablo R Ros,et al.  Motivation and compensation in academic radiology. , 2004, Journal of the American College of Radiology : JACR.

[94]  Dirk Helbing,et al.  How to create an innovation accelerator , 2010, ArXiv.

[95]  Michael Roach,et al.  A Taste for Science? PhD Scientists’ Academic Orientation and Self-Selection into Research Careers in Industry , 2010 .

[96]  Ernst Fehr,et al.  Fairness and Incentives in a Multi-Task Principal-Agent Model , 2004 .

[97]  J. Miner.  Commentary on Arthur Bedeian’s “The Manuscript Review Process: The Proper Roles of Authors, Referees, and Editors” , 2003 .

[98]  Teresa M. Amabile,et al.  Reward, Intrinsic Motivation, and Creativity , 1998 .

[99]  G. Glass,et al.  High-stakes testing and student achievement: Does accountability pressure increase student learning? , 2006 .

[100]  B. Frey,et al.  Motivation, Knowledge Transfer, and Organizational Forms , 1999 .

[101]  R. Nelson.  The market economy, and the scientific commons , 2004 .

[102]  R. Horton,et al.  How should medical science change? , 2014, The Lancet.

[103]  Paul R. Milgrom,et al.  Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design , 1991 .

[104]  Donald S. Siegel,et al.  A Theoretical and Empirical Analysis of Journal Rankings: The Case of Formal Lists , 2000 .

[105]  Burt Perrin,et al.  Effective Use and Misuse of Performance Measurement , 1998 .

[106]  Teresa M. Amabile,et al.  How to kill creativity. , 1998, Harvard business review.

[107]  B. Frey,et al.  Corporate Governance: What Can We Learn from Public Governance? , 2006 .

[108]  B. Frey.  Not Just for the Money: An Economic Theory of Personal Motivation , 1998 .

[109]  Marjori Matzke,et al.  F1000Prime recommendation of An index to quantify an individual's scientific research output. , 2005 .

[110]  D. Laband,et al.  Using and interpreting journal rankings: Introduction , 2013 .

[111]  F. Al-Shamali,et al.  Author Biographies. , 2015, Journal of social work in disability & rehabilitation.

[112]  Kathleen M. Eisenhardt,et al.  Control: Organizational and Economic Approaches , 1985 .

[113]  D. Cicchetti.  The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation , 1991, Behavioral and Brain Sciences.

[114]  F. Hayek,et al.  The Fatal Conceit: The Errors of Socialism , 1989 .

[115]  William H. Starbuck,et al.  How Much Better are the Most Prestigious Journals? The Statistics of Academic Publication , 2005, Organ. Sci..

[116]  M. Mahoney.  Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[117]  Eliot Freidson,et al.  Professionalism: The Third Logic , 2001 .

[118]  R. Merton,et al.  The Sociology of Science: Theoretical and Empirical Investigations , 1973 .

[119]  C. Donovan.  The qualitative future of research evaluation , 2007 .

[120]  Yudhijit Bhattacharjee,et al.  Citation impact. Saudi universities offer cash in exchange for academic prestige. , 2011, Science.

[121]  Eugene Garfield,et al.  Editors are justified in asking authors to cite equivalent references from same journal , 1997 .

[122]  Joel A. C. Baum,et al.  Free-Riding on Power Laws: questioning the validity of the Impact Factor as a measure of research quality in organization studies , 2011 .

[123]  F. Nijhuis,et al.  Prediction of scientific performance in clinical medicine , 1990 .

[124]  Mei-Shu Lai,et al.  The unintended consequence of diabetes mellitus pay-for-performance (P4P) program in Taiwan: are patients with more comorbidities or more severe conditions likely to be excluded from the P4P program? , 2011, Health services research.

[125]  J. Armstrong,et al.  Peer review for journals: Evidence on quality control, fairness, and innovation , 1997 .

[126]  Juan Miguel Campanario,et al.  Peer Review for Journals as it Stands Today—Part 2 , 1998 .

[127]  Lutz Bornmann,et al.  OPEN PEN ACCESS CCESS , 2008 .

[128]  Kevin G. Corley,et al.  Being Good Versus Looking Good: Business School Rankings and the Circean Transformation From Substance to Image , 2002 .

[129]  Peter Woelert,et al.  The ‘Economy of Memory’: Publications, Citations, and the Paradox of Effective Research Governance , 2013 .

[130]  Juan Miguel Campanario,et al.  UsingCitation Classics to study the incidence of serendipity in scientific discovery , 1996, Scientometrics.

[131]  E. Garfield.  Citation analysis as a tool in journal evaluation. , 1972, Science.

[132]  J. Kay.  Obliquity: Why Our Goals Are Best Achieved Indirectly , 2010 .

[133]  J. Lane.  Let's make science metrics more scientific , 2010, Nature.

[134]  M. Abernethy,et al.  Management control systems in research and development organizations: The role of accounting, behavior and personnel controls , 1997 .

[135]  Stefan Thurner,et al.  Peer-review in a world with rational scientists: Toward selection of the average , 2010, 1008.4324.

[136]  Daniel A. Levinthal,et al.  Hoping for A to Z While Rewarding Only A: Complex Organizations and Multiple Goals , 2009, Organ. Sci..

[137]  R. Nelson.  The Simple Economics of Basic Scientific Research , 1959, Journal of Political Economy.

[138]  Richard A. L. Jones.  The production of knowledge. , 2008, Nature nanotechnology.

[139]  Kirsten Bregn.  Pay for performance in the public sector: the challenge of fairness , 2015 .

[140]  Harvey Einbinder,et al.  International Encyclopedia of the Social Sciences , 2008 .

[141]  R. Posner.  From the new institutional economics to organization economics: with applications to corporate governance, government agencies, and legal institutions , 2010, Journal of Institutional Economics.

[142]  M. Schweitzer,et al.  Goals Gone Wild: The Systematic Side Effects of Overprescribing Goal Setting , 2009 .

[143]  Amanda H. Goodall.  Highly Cited Leaders and the Performance of Research Universities. , 2008 .

[144]  E M Selberg.  Creativity in Science. , 1962, Science.

[145]  P. Perakakis,et al.  The siege of science , 2008 .

[146]  Donald Gillies.  How Should Research Be Organised , 2008 .

[147]  M. V. Simkin,et al.  Copied citations create renowned papers , 2003, cond-mat/0305150.

[148]  Vincent Larivière,et al.  History of the journal impact factor: Contingencies and consequences , 2009, Scientometrics.

[149]  B. Frey,et al.  Do Rankings Reflect Research Quality? , 2008, SSRN Electronic Journal.

[150]  David V. Pritchett.  Econometric policy evaluation: A critique , 1976 .

[151]  M. Osterloh,et al.  Pay for Performance in the Public Sector—Benefits and (Hidden) Costs , 2010 .

[152]  Bruno S. Frey,et al.  Tertium Datur: Pricing, Regulating and Intrinsic Motivation , 1992 .