Improving transparency and scientific rigor in academic publishing

Progress in basic and clinical research is slowed when researchers fail to provide a complete and accurate report of how a study was designed, executed, and the results analyzed. Publishing rigorous scientific research involves a full description of the methods, materials, procedures, and outcomes. Investigators may fail to provide a complete description of how their study was designed and executed because they may not know how to accurately report the information or the mechanisms are not in place to facilitate transparent reporting. Here, we provide an overview of how authors can write manuscripts in a transparent and thorough manner. We introduce a set of reporting criteria that can be used for publishing, including recommendations on reporting the experimental design and statistical approaches. We also discuss how to accurately visualize the results and provide recommendations for peer reviewers to enhance rigor and transparency. Incorporating transparency practices into research manuscripts will significantly improve the reproducibility of the results by independent laboratories.

[1]  David P Wolfer,et al.  Similar reliability and equivalent performance of female and male mice in the open field and water‐maze place navigation task , 2017, American journal of medical genetics. Part C, Seminars in medical genetics.

[2]  Karl P. Pfeiffer,et al.  The Use of Statistics in Medical Research , 2007 .

[3]  Stanley E. Lazic,et al.  Experimental Design for Laboratory Biologists: Maximising Information and Improving Reproducibility. By Stanley E. Lazic. Cambridge and New York: Cambridge University Press. $175.00 (hardcover); $64.99 (paper). xv + 412 p.; ill.; index. ISBN: 978-1-107-07429-3 (hc); 978-1-107-42488-3 (pb); 978-1-316 , 2017, The Quarterly Review of Biology.

[4]  Jeffrey D. Scargle,et al.  Publication Bias: The “File-Drawer” Problem in Scientific Inference , 2000 .

[5]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[6]  Katja Jasinskaja,et al.  Elaboration and Explanation ⋆ , 2011 .

[7]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[8]  A Rhumba of “R’s”: Replication, Reproducibility, Rigor, Robustness: What Does a Failure to Replicate Mean? , 2016, eNeuro.

[9]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[10]  J. Clayton,et al.  Applying the new SABV (sex as a biological variable) policy to research and clinical care , 2017, Physiology & Behavior.

[11]  M. Lauer,et al.  Reproducibility 2020 : Progress and priorities , 2017 .

[12]  David N. Kennedy,et al.  The Resource Identification Initiative: A cultural shift in publishing , 2015, Neuroinformatics.

[13]  Megan A. K. Peters,et al.  Perceptual confidence neglects decision-incongruent evidence in the brain , 2017, Nature Human Behaviour.

[14]  Reporters find science journals harder to trust, but not easy to verify. , 2006, The New York times on the Web.

[15]  Alex R. Smith,et al.  Sex differences in the structural connectome of the human brain , 2013, Proceedings of the National Academy of Sciences.

[16]  U. Dirnagl,et al.  Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke , 2016, PLoS biology.

[17]  Anne Cambon-Thomsen,et al.  Developing a guideline to standardize the citation of bioresources in journal articles (CoBRA) , 2015, BMC Medicine.

[18]  Maryann E Martone,et al.  The Resource Identification Initiative: a cultural shift in publishing , 2015, Brain and behavior.

[19]  S. Goodman,et al.  The Use of Predicted Confidence Intervals When Planning Experiments and the Misuse of Power When Interpreting Results , 1994, Annals of Internal Medicine.

[20]  A. Coats,et al.  Disclosure of negative trial results. A call for action. , 2015, International journal of cardiology.

[21]  F. Godlee,et al.  Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. , 1998, JAMA.

[22]  B. Cohen,et al.  How should novelty be valued in science? , 2017, eLife.

[23]  David Ross,et al.  Randomization, blinding, and coding , 2015 .

[24]  Stanley E Lazic,et al.  Four simple ways to increase power without increasing the sample size , 2018, Laboratory animals.

[25]  Nicole A. Vasilevsky,et al.  On the reproducibility of science: unique identification of research resources in the biomedical literature , 2013, PeerJ.

[26]  J. Clayton.  Studying both sexes: a guiding principle for biomedicine , 2016, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[27]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[28]  S. Lazic,et al.  A call for transparent reporting to optimize the predictive value of preclinical research , 2012, Nature.

[29]  Nancy L. Leech,et al.  Post-Hoc Power: A Concept Whose Time Has Come. , 2004 .

[30]  Regina Nuzzo,et al.  Scientific method: Statistical errors , 2014, Nature.

[31]  Amanda Capes-Davis,et al.  Authentication: A Standard Problem or a Problem of Standards? , 2016, PLoS biology.

[32]  S. Muthukumaraswamy,et al.  Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond , 2014 .

[33]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[34]  Avijit Hazra,et al.  How to choose the right statistical test? , 2011, Indian journal of ophthalmology.

[35]  Guillaume A. Rousselet,et al.  A few simple steps to improve the description of group results in neuroscience , 2016, The European journal of neuroscience.

[36]  Ulrich Dirnagl,et al.  Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation , 2014, PLoS biology.

[37]  Published Online.  Biomedical research: increasing value, reducing waste , 2014 .

[38]  M. H. Ensom,et al.  Post Hoc Power Analysis: An Idea Whose Time Has Passed? , 2001, Pharmacotherapy.

[39]  Douglas G. Altman,et al.  Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research , 2010, Journal of pharmacology & pharmacotherapeutics.

[40]  M. Ritskes-Hoitinga,et al.  A Gold Standard Publication Checklist to Improve the Quality of Animal Studies, to Fully Integrate the Three Rs, and to Make Systematic Reviews More Feasible , 2010, Alternatives to laboratory animals : ATLA.

[41]  V. Garovic,et al.  Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm , 2015, PLoS biology.

[42]  Harvey J Motulsky,et al.  Common Misconceptions about Data Analysis and Statistics , 2014, The Journal of Pharmacology and Experimental Therapeutics.

[43]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[44]  L. Cahill,et al.  Sex differences in molecular neuroscience: from fruit flies to humans , 2010, Nature Reviews Neuroscience.

[45]  D F Thompson,et al.  Understanding financial conflicts of interest. , 1993, The New England journal of medicine.

[46]  I. Cuthill,et al.  Survey of the Quality of Experimental Design, Statistical Analysis and Reporting of Research Using Animals , 2009, PloS one.

[47]  Joshua K. Hartshorne,et al.  Tracking Replicability as a Method of Post-Publication Open Evaluation , 2011, Front. Comput. Neurosci..

[48]  Kp Suresh.  An overview of randomization techniques: An unbiased assessment of outcome in clinical research , 2011, Journal of human reproductive sciences.

[49]  Douglas G Altman,et al.  Guidelines for the design and statistical analysis of experiments using laboratory animals. , 2002, ILAR journal.

[50]  Gillian L. Currie,et al.  Risk of Bias in Reports of In Vivo Research: A Focus for Improvement , 2015, PLoS biology.

[51]  I. Cuthill,et al.  Animal Research: Reporting In Vivo Experiments: The ARRIVE Guidelines , 2010, British journal of pharmacology.

[52]  Zheng Su,et al.  The pursuit of balance: An overview of covariate-adaptive randomization techniques in clinical trials. , 2015, Contemporary clinical trials.

[53]  Kenneth F Schulz,et al.  Blinding in randomised trials: hiding who got what , 2002, The Lancet.

[54]  P. Nagele,et al.  Misuse of standard error of the mean (SEM) when reporting variability of a sample. A critical evaluation of four anaesthesia journals. , 2003, British journal of anaesthesia.

[55]  M. Macleod,et al.  Preclinical studies of human disease: time to take methodological quality seriously. , 2011, Journal of molecular and cellular cardiology.

[56]  A. Arnold,et al.  Incorporating sex as a biological variable in neuroscience: what do we gain? , 2017, Nature Reviews Neuroscience.

[57]  John P. A. Ioannidis,et al.  Research: increasing value, reducing waste 2 , 2014 .

[58]  Katie Lidster,et al.  Two Years Later: Journals Are Not Yet Enforcing the ARRIVE Guidelines on Reporting Standards for Pre-Clinical Animal Studies , 2014, PLoS biology.

[59]  R. Stevens,et al.  Bias in the reporting of sex and age in biomedical research on mouse models , 2016, eLife.

[60]  Douglas G. Altman,et al.  Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) , 2015, Circulation.

[61]  Natalie Matosin,et al.  Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture , 2014, Disease Models & Mechanisms.

[62]  Guillaume A Rousselet,et al.  A Guide to Robust Statistical Methods in Neuroscience , 2017, bioRxiv.

[63]  Mohini P. Barde,et al.  What to use to express the variability of data: Standard deviation or standard error of mean? , 2012, Perspectives in clinical research.

[64]  Eric M Prager,et al.  Transparent reporting for reproducible science , 2016, Journal of neuroscience research.

[65]  Kenneth M. Yamada,et al.  Reproducibility and cell biology , 2015, The Journal of cell biology.

[66]  Sally Galbraith,et al.  A Study of Clustered Data and Approaches to Its Analysis , 2010, The Journal of Neuroscience.

[67]  J. Carpenter,et al.  Effects of training on quality of peer review: randomised controlled trial , 2004, BMJ : British Medical Journal.

[68]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[69]  Machelle D. Wilson,et al.  Valid statistical approaches for analyzing sholl data: Mixed effects versus simple linear models , 2017, Journal of Neuroscience Methods.

[70]  L. Freedman,et al.  Reproducibility2020: Progress and priorities , 2017, bioRxiv.

[71]  N. Grunberg,et al.  The Importance of Reporting Housing and Husbandry in Rat Research , 2011, Front. Behav. Neurosci..

[72]  J. Hoenig,et al.  Statistical Practice The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis , 2001 .

[73]  Christina K. Pikas,et al.  A multi-disciplinary perspective on emergent and future innovations in peer review , 2017, F1000Research.

[74]  I. Segarra,et al.  Sex-Divergent Clinical Outcomes and Precision Medicine: An Important New Role for Institutional Review Boards and Research Ethics Committees , 2017, Front. Pharmacol..

[75]  I. Cuthill,et al.  Reporting : The ARRIVE Guidelines for Reporting Animal Research , 2010 .

[76]  Leif D. Nelson,et al.  Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” , 2014 .

[77]  E. Wagenmakers,et al.  Detecting and avoiding likely false‐positive findings – a practical guide , 2017, Biological reviews of the Cambridge Philosophical Society.

[78]  J. Crabbe,et al.  Genetics of mouse behavior: interactions with laboratory environment. , 1999, Science.

[79]  D. Moher,et al.  Four Proposals to Help Improve the Medical Research Literature , 2015, PLoS medicine.

[80]  David Moher,et al.  Reducing waste from incomplete or unusable reports of biomedical research , 2014, The Lancet.

[81]  I. Cockburn,et al.  The Economics of Reproducibility in Preclinical Research , 2015, PLoS biology.

[82]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[83]  S. Young.  Bias in the research literature and conflict of interest: an issue for publishers, editors, reviewers and authors, and it is not just about the money. , 2009, Journal of psychiatry & neuroscience : JPN.

[84]  S. Goodman,et al.  Reproducible Research: Moving toward Research the Public Can Really Trust , 2007, Annals of Internal Medicine.

[85]  Mohit Bhandari,et al.  Practical tips for surgical research: blinding: who, what, when, why, how? , 2010, Canadian journal of surgery. Journal canadien de chirurgie.

[86]  Arturo Casadevall,et al.  Why Has the Number of Scientific Retractions Increased? , 2013, PloS one.

[87]  A H Smith,et al.  Confidence limit analyses should replace power calculations in the interpretation of epidemiologic studies. , 1992, Epidemiology.

[88]  A. Beery,et al.  Inclusion of females does not increase variability in rodent research studies , 2018, Current Opinion in Behavioral Sciences.

[89]  Marko Savic,et al.  Data visualization, bar naked: A free tool for creating interactive graphics , 2017, The Journal of Biological Chemistry.

[90]  Harold Varmus,et al.  Rescuing US biomedical research from its systemic flaws , 2014, Proceedings of the National Academy of Sciences.

[91]  Marko Savic,et al.  Correction: From Static to Interactive: Transforming Data Visualization to Improve Transparency , 2016, PLoS biology.

[92]  K. Dickersin,et al.  Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. , 1992, JAMA.

[93]  Stanley E Lazic,et al.  What exactly is ‘N’ in cell culture and animal experiments? , 2017, bioRxiv.

[94]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[95]  Douglas G Altman,et al.  Standard deviations and standard errors , 2005, BMJ : British Medical Journal.

[96]  Julia Kastner,et al.  Introduction to Robust Estimation and Hypothesis Testing , 2005 .

[97]  Christian Gluud,et al.  Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? , 2003, JAMA.

[98]  John J Dziak,et al.  Factorial experiments: efficient tools for evaluation of intervention components. , 2014, American journal of preventive medicine.

[99]  Stanley E Lazic,et al.  The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? , 2010, BMC Neuroscience.

[100]  Kalina Trenevska Blagoeva.  Factorial Experiments , 2011, International Encyclopedia of Statistical Science.

[101]  Freek E. Hoebeek,et al.  SLC26A11 (KBAT) in Purkinje Cells Is Critical for Inhibitory Transmission and Contributes to Locomotor Coordination , 2016, eNeuro.

[102]  P. Shekelle,et al.  Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation , 2015, BMJ : British Medical Journal.

[103]  V. Garovic,et al.  Reinventing Biostatistics Education for Basic Scientists , 2016, PLoS biology.