The credibility crisis in research: Can economics tools help?

The issue of nonreplicable evidence has attracted considerable attention across biomedical and other sciences. This concern is accompanied by an increasing interest in reforming research incentives and practices. How to optimally perform these reforms is a scientific problem in itself, and economics has several scientific methods that can help evaluate research reforms. Here, we review these methods and show their potential. Prominent among them are mathematical modeling and laboratory experiments that constitute affordable ways to approximate the effects of policies with wide-ranging implications.

[1]  Jelte M. Wicherts,et al.  Researchers’ Intuitions About Power in Psychological Research , 2016, Psychological science.

[2]  Avid,et al.  Experiments Testing Multiobject Allocation Mechanisms , 2003 .

[3]  Carl T. Bergstrom,et al.  Publication bias and the canonization of false facts , 2016, eLife.

[4]  J. Ioannidis.  Why Science Is Not Necessarily Self-Correcting , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[5]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[6]  John P. A. Ioannidis,et al.  How to Make More Published Research True , 2014, PLoS medicine.

[7]  P. Sørensen,et al.  Persuasion Bias in Science: Can Economics Help? , 2016 .

[8]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[9]  G. Wilkinson,et al.  Open peer review: A randomised controlled trial , 2000, British Journal of Psychiatry.

[10]  M. Ottaviani,et al.  Research and the Approval Process , 2014 .

[11]  Raj Chetty,et al.  What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics , 2014 .

[12]  Charles R. Plott,et al.  A Pari-Mutuel-Like Mechanism for Information Aggregation: A Field Test inside Intel , 2014, Journal of Political Economy.

[13]  Armin Falk,et al.  Representative evidence on lying costs , 2014 .

[14]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[15]  Marcus R. Munafò,et al.  Modelling the effects of subjective and objective decision making in scientific peer review , 2013, Nature.

[16]  Richard McElreath,et al.  The natural selection of bad science , 2016, Royal Society Open Science.

[17]  Sanford J. Grossman.  The Informational Role of Warranties and Private Disclosure about Product Quality , 1981, The Journal of Law and Economics.

[18]  Susann Fiedler,et al.  Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency , 2016, PLoS biology.

[19]  L. Trinquart,et al.  Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis , 2016, BMC Medicine.

[20]  Richard A. Bettis,et al.  The search for asterisks: Compromised statistical tests and flawed theories , 2012 .

[21]  L. Zirulia,et al.  The Economics of Scientific Misconduct , 2008 .

[22]  Tony Delamothe,et al.  Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial , 2010, BMJ : British Medical Journal.

[23]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[24]  Paul E. Smaldino,et al.  Replication, Communication, and the Population Dynamics of Scientific Discovery , 2015, PloS one.

[25]  Charles R. Plott,et al.  Market architectures, institutional landscapes and testbed experiments , 1994 .

[26]  D. Rennie,et al.  Guarding the guardians: a conference on editorial peer review. , 1986, JAMA.

[27]  M. Dufwenberg,et al.  Working Papers in Economics and Statistics Deception through Telling the Truth?! Experimental Evidence from Individuals and Teams Deception through Telling the Truth?! Experimental Evidence from Individuals and Teams , 2022 .

[28]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[29]  Phillip Li,et al.  Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not' , 2015 .

[30]  L. S. Shapley,et al.  College Admissions and the Stability of Marriage , 2013, Am. Math. Mon..

[31]  R. Blank.  The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review , 1991 .

[32]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, BMC medicine.

[33]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[34]  Georges Annas.  The researcher's dilemma , 1995, Nature.

[35]  Timothy N. Cason,et al.  Market power in tradable emission markets: a laboratory testbed for emission trading in Port Phillip Bay, Victoria , 2003 .

[36]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[37]  J. Kagel,et al.  The Dynamics of Reorganization in Matching Markets: A Laboratory Experiment Motivated by a Natural Experiment , 2000 .

[38]  David Moher,et al.  CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials , 2010, PLoS medicine.

[39]  T. Kealey,et al.  Modelling science as a contribution good , 2014 .

[40]  Erin A. Cech,et al.  How Economics Shapes Science , 2013 .

[41]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[42]  Thomas Gall,et al.  Evaluating solutions to the problem of false positives , 2019, Research Policy.

[43]  Marcus R. Munafò,et al.  Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions , 2016, PLoS biology.

[44]  R. Hertwig,et al.  Experimental practices in economics: A methodological challenge for psychologists? , 2001, Behavioral and Brain Sciences.

[45]  Paul R. Milgrom,et al.  Good News and Bad News: Representation Theorems and Applications , 1981 .

[46]  Dan Ariely,et al.  The Dark Side of Creativity : Original Thinkers Can Be More Dishonest , 2013 .

[47]  B. Frey,et al.  The Cost of Price Incentives: An Empirical Analysis of Motivation Crowding-Out , 1997 .

[48]  A. Roth.  The Economist as Engineer: Game Theory, Experimentation, and Computation as Tools for Design Economics , 2002 .

[49]  D. Ariely,et al.  PSYCHOLOGICAL SCIENCE Research Article Contagion and Differentiation in Unethical Behavior The Effect of One Bad Apple on the Barrel , 2022 .

[50]  David Card,et al.  Page Limits on Economics Articles: Evidence from Two Journals , 2014 .

[51]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[52]  S. Lazic,et al.  A call for transparent reporting to optimize the predictive value of preclinical research , 2012, Nature.

[53]  John P A Ioannidis,et al.  Stealth research: is biomedical innovation happening outside the peer-reviewed literature? , 2015, JAMA.

[54]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[55]  U. Fischbacher,et al.  Lies in Disguise. An experimental study on cheating , 2013 .

[56]  Christopher M. Snyder,et al.  Open Access Versus Traditional Journal Pricing: Using a Simple 'Platform Market' Model to Understand Which Will Win (and Which Should) , 2012 .

[57]  J. Abeler,et al.  Preferences for Truth-Telling , 2016, Econometrica.

[58]  Joshua S. Gans,et al.  Why Referees Are Not Paid (Enough) , 1998 .

[59]  Daniele Fanelli,et al.  Redefine misconduct as distorted reporting , 2013, Nature.

[60]  Philip Kitcher,et al.  The Division of Cognitive Labor , 1990 .

[61]  Paul R. Milgrom,et al.  What the Seller Wont Tell You: Persuasion and Disclosure in Markets , 2009 .

[62]  Abel Brodeur,et al.  Star Wars: The Empirics Strike Back , 2012, SSRN Electronic Journal.

[63]  Uri Gneezy,et al.  Deception: The Role of Consequences , 2005 .

[64]  Elisabeth Schulte,et al.  Strategic Private Experimentation , 2014 .

[65]  Emeric Henry.  Strategic Disclosure of Research Results: The Cost of Proving Your Honesty , 2009 .

[66]  P. David,et al.  Toward a new economics of science , 1994 .

[67]  B. Rockenbach,et al.  Measuring lying aversion , 2013 .