Identifying and Overcoming Threats to Reproducibility, Replicability, Robustness, and Generalizability in Microbiome Research

ABSTRACT The “reproducibility crisis” in science affects microbiology as much as any other area of inquiry, and microbiologists have long struggled to make their research reproducible. We need to respect that ensuring that our methods and results are sufficiently transparent is difficult. This difficulty is compounded in interdisciplinary fields such as microbiome research. There are many reasons why a researcher is unable to reproduce a previous result, and even if a result is reproducible, it may not be correct. Furthermore, failures to reproduce previous results have much to teach us about the scientific process and microbial life itself. This Perspective delineates a framework for identifying and overcoming threats to reproducibility, replicability, robustness, and generalizability of microbiome research. Instead of seeing signs of a crisis in others’ work, we need to appreciate the technical and social difficulties that limit reproducibility in the work of others as well as our own.

[1]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[2]  Arturo Casadevall,et al.  A Framework for Improving the Quality of Research in the Biological Sciences , 2016, mBio.

[3]  Elisabeth Pain.  Register your study as a new publication option , 2015 .

[4]  R Balfour Sartor,et al.  Specific microbiota direct the differentiation of IL-17-producing T-helper cells in the mucosa of the small intestine. , 2008, Cell host & microbe.

[5]  Qing Guo,et al.  A systematic review of the reporting of sample size calculations and corresponding data components in observational functional magnetic resonance imaging studies , 2014, NeuroImage.

[6]  Whitaker Kirstie.  Publishing a reproducible paper , 2017 .

[7]  Pelin Yilmaz,et al.  The SILVA and “All-species Living Tree Project (LTP)” taxonomic frameworks , 2013, Nucleic Acids Res..

[8]  Jeffrey T. Leek,et al.  Opinion: Reproducible research can still be wrong: Adopting a prevention approach , 2015, Proceedings of the National Academy of Sciences.

[9]  Brian A. Nosek,et al.  An open investigation of the reproducibility of cancer biology research , 2014, eLife.

[10]  H. Van de Sompel,et al.  Scholarly Context Not Found: One in Five Articles Suffers from Reference Rot , 2014, PloS one.

[11]  Elisabeth Pain.  After the bombs. , 2015, Science.

[12]  Jacques Ravel,et al.  “Available upon request”: not good enough for microbiome data! , 2018, Microbiome.

[13]  Yihui Xie,et al.  Dynamic Documents with R and knitr, Second Edition , 2015 .

[14]  F. Collins,et al.  NIH plans to enhance reproducibility , 2014 .

[15]  Philip E. Bourne,et al.  The Mycobacterium tuberculosis Drugome and Its Polypharmacological Implications , 2010, PLoS Comput. Biol..

[16]  J. Walter,et al.  A critical assessment of the “sterile womb” and “in utero colonization” hypotheses: implications for research on the pioneer infant microbiome , 2017, Microbiome.

[17]  Kun Huang,et al.  Injury, Large-Scale Collaborative Research Program , 2015 .

[18]  David M. Simcha,et al.  Tackling the widespread and critical impact of batch effects in high-throughput data , 2010, Nature Reviews Genetics.

[19]  Pamela C. Ronald,et al.  Retraction: Small Protein-Mediated Quorum Sensing in a Gram-Negative Bacterium , 2013, PloS one.

[20]  et al.,et al.  Jupyter Notebooks - a publishing format for reproducible computational workflows , 2016, ELPUB.

[21]  P. Schloss,et al.  Dynamics and associations of microbial community types across the human body , 2014, Nature.

[22]  William Stafford Noble.  A Quick Guide to Organizing Computational Biology Projects , 2009, PLoS Comput. Biol..

[23]  E. Mardis,et al.  An obesity-associated gut microbiome with increased capacity for energy harvest , 2006, Nature.

[24]  Dan R. Littman,et al.  Induction of Intestinal Th17 Cells by Segmented Filamentous Bacteria , 2009, Cell.

[25]  Greg Wilson,et al.  Software Carpentry: lessons learned , 2013, F1000Research.

[26]  C. Huttenhower,et al.  Experimental design and quantitative analysis of microbial community multiomics , 2017, Genome Biology.

[27]  A. Gelman,et al.  The statistical crisis in science , 2014 .

[28]  Patrick D. Schloss,et al.  Looking for a Signal in the Noise: Revisiting Obesity and the Microbiome , 2016, mBio.

[29]  D. Bessesen,et al.  Human gut microbes associated with obesity , 2007 .

[30]  F. Bäckhed,et al.  Obesity alters gut microbial ecology. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[31]  Kai Blin,et al.  Ten Simple Rules for Taking Advantage of Git and GitHub , 2014, bioRxiv.

[32]  Eoin L. Brodie,et al.  Greengenes, a Chimera-Checked 16S rRNA Gene Database and Workbench Compatible with ARB , 2006, Applied and Environmental Microbiology.

[33]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[34]  P. Turnbaugh,et al.  Microbial ecology: Human gut microbes associated with obesity , 2006, Nature.

[35]  Patrick D. Schloss,et al.  Clostridium difficile Colonizes Alternative Nutrient Niches during Infection across Distinct Murine Gut Microbiomes , 2017, mSystems.

[36]  Emily R. Davenport,et al.  Genetic Determinants of the Gut Microbiome in UK Twins. , 2016, Cell host & microbe.

[37]  Morgan Taschuk,et al.  Ten simple rules for making research software more robust , 2016, PLoS Comput. Biol..

[38]  B. Roe,et al.  A core gut microbiome in obese and lean twins , 2008, Nature.

[39]  Morris A. Swertz,et al.  Population-based metagenomics analysis reveals markers for gut microbiome composition and diversity , 2016, Science.

[40]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[41]  Andrew Gelman,et al.  Why We (Usually) Don't Have to Worry About Multiple Comparisons , 2009, 0907.2478.

[42]  Greg Wilson,et al.  Software Carpentry: lessons learned , 2014, F1000Research.

[43]  F. Bushman,et al.  Comparison of placenta samples with contamination controls does not provide evidence for a distinct placenta microbiota , 2016, Microbiome.

[44]  Victoria Stodden,et al.  An empirical analysis of journal policy effectiveness for computational reproducibility , 2018, Proceedings of the National Academy of Sciences.

[45]  Martin Hartmann,et al.  Introducing mothur: Open-Source, Platform-Independent, Community-Supported Software for Describing and Comparing Microbial Communities , 2009, Applied and Environmental Microbiology.

[46]  Pieter C. Dorrestein,et al.  Multi-omics Analysis of Periodontal Pocket Microbial Communities Pre- and Posttreatment , 2017, mSystems.

[47]  M. Munafo,et al.  Robust research needs many lines of evidence , 2018, Nature.

[48]  Jacques Ravel,et al.  All hail reproducibility in microbiome research , 2014, Microbiome.

[49]  N. Lane.  The unseen world: reflections on Leeuwenhoek (1677) ‘Concerning little animals’ , 2015, Philosophical Transactions of the Royal Society B: Biological Sciences.

[50]  Katherine H. Huang,et al.  Structure, Function and Diversity of the Healthy Human Microbiome , 2012, Nature.

[51]  John Chilton,et al.  Common Workflow Language, v1.0 , 2016 .

[52]  Jeroen Raes,et al.  How informative is the mouse for human gut microbiota research? , 2015, Disease Models & Mechanisms.

[53]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[54]  The importance of being second , 2018, PLoS biology.

[55]  R. Knight,et al.  Meta‐analyses of human gut microbes associated with obesity and IBD , 2014, FEBS letters.

[56]  S. Horbach,et al.  The ghosts of HeLa: How cell line misidentification contaminates the scientific literature , 2017, PloS one.

[57]  James R. Cole,et al.  Ribosomal Database Project: data and tools for high throughput rRNA analysis , 2013, Nucleic Acids Res..

[58]  Yihui Xie,et al.  Dynamic Documents with R and knitr , 2015 .

[59]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[60]  Patrick D Schloss,et al.  Comparison of the respiratory microbiome in healthy nonsmokers and smokers. , 2013, American journal of respiratory and critical care medicine.

[61]  W. J. Hadden,et al.  A Comparison of , 1971 .

[62]  J. Leek,et al.  What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[63]  A. Casadevall,et al.  Reproducible Science , 2010, Infection and Immunity.

[64]  T. Miyakawa,et al.  Genomic responses in mouse models poorly mimic human inflammatory diseases , 2013 .

[65]  Young-Su Seo,et al.  Retraction. A type I-secreted, sulfated peptide triggers XA21-mediated innate immunity. , 2013, Science.

[66]  R. Gamelli,et al.  Genomic responses in mouse models poorly mimic human inflammatory diseases , 2013, Proceedings of the National Academy of Sciences.

[67]  Louette R. Johnson Lutjens.  Research , 2006 .

[68]  Arturo Casadevall,et al.  Promoting Responsible Scientific Research , 2015 .

[69]  R. Knight,et al.  Analysis of the Gut Microbiota in the Old Order Amish and Its Relation to the Metabolic Syndrome , 2012, PloS one.

[70]  Helen Pearson.  Competition in biology: It's a scoop! , 2003, Nature.

[71]  Yolanda Gil,et al.  Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome , 2013, PloS one.

[72]  Kyle Bittinger,et al.  Optimizing methods and dodging pitfalls in microbiome research , 2017, Microbiome.

[73]  Paul Turner,et al.  Reagent and laboratory contamination can critically impact sequence-based microbiome analyses , 2014, BMC Biology.

[74]  J. Vandekerckhove,et al.  A Bayesian Perspective on the Reproducibility Project: Psychology , 2016, PloS one.

[75]  Yaqing Huang,et al.  Investigation of Cross-Contamination and Misidentification of 278 Widely Used Tumor Cell Lines , 2017, PloS one.

[76]  François Michonneau,et al.  Ten Simple Rules for Digital Data Storage , 2016, PeerJ Prepr..

[77]  Timothy J. Laurent,et al.  A Taxonomic Signature of Obesity in the Microbiome? Getting to the Guts of the Matter , 2014, PloS one.

[78]  J. Raes,et al.  Heterogeneity of the gut microbiome in mice: guidelines for optimizing experimental design , 2015, FEMS microbiology reviews.

[79]  R. Lanfear,et al.  The Extent and Consequences of P-Hacking in Science , 2015, PLoS biology.