Modelling science trustworthiness under publish or perish pressure

Scientific publication is immensely important to the scientific endeavour. There is, however, concern that rewarding scientists chiefly on publication creates a perverse incentive, allowing careless and fraudulent conduct to thrive, compounded by the predisposition of top-tier journals towards novel, positive findings rather than investigations confirming null hypothesis. This potentially compounds a reproducibility crisis in several fields, and risks undermining science and public trust in scientific findings. To date, there has been comparatively little modelling on factors that influence science trustworthiness, despite the importance of quantifying the problem. We present a simple phenomenological model with cohorts of diligent, careless and unethical scientists, with funding allocated by published outputs. This analysis suggests that trustworthiness of published science in a given field is influenced by false positive rate, and pressures for positive results. We find decreasing available funding has negative consequences for resulting trustworthiness, and examine strategies to combat propagation of irreproducible science.

[1]  Melissa S. Anderson,et al.  Normal Misbehavior: Scientists Talk about the Ethics of Research , 2006, Journal of empirical research on human research ethics : JERHRE.

[2]  COPE: committee on publication ethics , 2000, The British journal of surgery.

[3]  Jatinder Singh,et al.  Committee on publication ethics , 2010, Journal of pharmacology & pharmacotherapeutics.

[4]  Martin Krzywinski,et al.  Points of significance: P values and the search for significance , 2016, Nature Methods.

[5]  P. Lawrence The politics of publication , 2003, Nature.

[6]  M. Baker 1,500 scientists lift the lid on reproducibility , 2016, Nature.

[7]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[8]  Simone Rödder,et al.  Fraud: causes and culprits as perceived by science and the media , 2007, EMBO reports.

[9]  Rustam Al-Shahi Salman,et al.  Increasing value and reducing waste in biomedical research regulation and management , 2014, The Lancet.

[10]  T. Opthof,et al.  Fraud and misconduct in science: the stem cell seduction , 2009, Netherlands heart journal : monthly journal of the Netherlands Society of Cardiology and the Netherlands Heart Foundation.

[11]  J Ranstam,et al.  Fraud in medical research: an international survey of biostatisticians. ISCB Subcommittee on Fraud. , 2000, Controlled clinical trials.

[12]  P. Lachenbruch,et al.  Fraud in medical research. , 2000, JAMA.

[13]  Michał Krawczyk,et al.  The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature , 2015, PloS one.

[14]  Charles W Lidz,et al.  Authors' reports about research integrity problems in clinical trials. , 2005, Contemporary clinical trials.

[15]  D. Curran‐Everett,et al.  The fickle P value generates irreproducible results , 2015, Nature Methods.

[16]  F. Islami,et al.  Multimorbidity: Epidemiology and Risk Factors in the Golestan Cohort Study, Iran , 2016, Medicine.

[17]  Paul E. Smaldino,et al.  Replication, Communication, and the Population Dynamics of Scientific Discovery , 2015, PloS one.

[18]  David van Dijk,et al.  Publication metrics and success on the academic job market , 2014, Current Biology.

[19]  L. Claxton,et al.  Scientific authorship. Part 1. A window into scientific fraud? , 2005, Mutation research.

[20]  M. Edwards,et al.  Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition , 2017, Environmental engineering science.

[21]  John P. A. Ioannidis,et al.  Assessing value in biomedical research: the PQRST of appraisal and reward. , 2014, JAMA.

[22]  J. Ioannidis,et al.  Reproducibility in Science: Improving the Standard for Basic and Preclinical Research , 2015, Circulation research.

[23]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[24]  J. Ioannidis,et al.  Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. , 2016, JAMA.

[25]  John P.A. Ioannidis,et al.  Inflated numbers of authors over time have not been just due to increasing research complexity. , 2008, Journal of clinical epidemiology.

[26]  A. Casadevall,et al.  Misconduct accounts for the majority of retracted scientific publications , 2012, Proceedings of the National Academy of Sciences.

[27]  N. Saunders Faculty Opinions recommendation of Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and hypercompetition. , 2018, Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature.

[28]  David Robert Grimes,et al.  On the Viability of Conspiratorial Beliefs , 2016, PloS one.

[29]  J. Glick Scientific data audit—A key management tool , 1992 .

[30]  D. Fanelli Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data , 2010, PloS one.

[31]  Maarten van Wesel,et al.  Evaluation by Citation: Trends in Publication Behavior, Evaluation Criteria, and the Strive for High Impact Publications , 2015, Science and Engineering Ethics.

[32]  Nicholas H. Steneck,et al.  Fostering integrity in research: Definitions, current knowledge, and future directions , 2006, Science and Engineering Ethics.

[33]  Fostering reproducible fMRI research , 2017, Nature Communications.

[34]  Seema Rawat,et al.  Publish or perish: Where are we heading? , 2014, Journal of research in medical sciences : the official journal of Isfahan University of Medical Sciences.

[35]  Lucy Carter,et al.  A case for a duty to feed the hungry: GM plants and the third world , 2007, Sci. Eng. Ethics.

[36]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[37]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[38]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[39]  Melissa S. Anderson,et al.  Scientists behaving badly , 2005, Nature.

[40]  John P. A. Ioannidis,et al.  How to Make More Published Research True , 2014, PLoS medicine.

[41]  Stuart J. Ritchie,et al.  Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect , 2012, PloS one.

[42]  Lutz Bornmann,et al.  Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references , 2014, J. Assoc. Inf. Sci. Technol..

[43]  Ushma S. Neill,et al.  Publish or perish, but at what cost? , 2008, The Journal of clinical investigation.

[44]  S. Fuchs,et al.  Fraud and Trust in Science , 2015, Perspectives in biology and medicine.

[45]  Richard McElreath,et al.  The natural selection of bad science , 2016, Royal Society Open Science.

[46]  D. Grimes Correction: On the Viability of Conspiratorial Beliefs , 2016, PloS one.

[47]  Y. Smulders,et al.  Publication Pressure and Burn Out among Dutch Medical Professors: A Nationwide Survey , 2013, PloS one.

[48]  Richard W. Willard The Nationwide Survey. , 1972 .

[49]  J. Ioannidis,et al.  Magnitude of effects in clinical trials published in high-impact general medical journals. , 2011, International journal of epidemiology.

[50]  David Colquhoun,et al.  An investigation of the false discovery rate and the misinterpretation of p-values , 2014, Royal Society Open Science.

[51]  John P. A. Ioannidis,et al.  Meta-assessment of bias in science , 2017, Proceedings of the National Academy of Sciences.

[52]  Harlan M Krumholz,et al.  Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis , 2012, BMJ : British Medical Journal.

[53]  D. Fanelli How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[54]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[55]  Megan Scudellari State of denial , 2010, Nature Network Boston.

[56]  Jim Giles,et al.  Special Report: Taking on the cheats , 2005, Nature.