A Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines

SUMMARY Evidence that some high-impact biomedical results cannot be repeated has stimulated interest in practices that generate findable, accessible, interoperable, and reusable (FAIR) data. Multiple papers have identified specific examples of irreproducibility, but practical ways to make data more reproducible have not been widely studied. Here, five research centers in the NIH LINCS Program Consortium investigate the reproducibility of a prototypical perturbational assay: quantifying the responsiveness of cultured cells to anti-cancer drugs. Such assays are important for drug development, studying cellular networks, and patient stratification. While many experimental and computational factors impact intra- and inter-center reproducibility, the factors most difficult to identify and control are those with a strong dependency on biological context. These factors often vary in magnitude with the drug being analyzed and with growth conditions. We provide ways to identify such context-sensitive factors, thereby improving both the theory and practice of reproducible cell-based assays.

[1]  Erik Schultes,et al.  The FAIR Guiding Principles for scientific data management and stewardship , 2016, Scientific Data.

[2]  Jose Russo,et al.  Human Breast Epithelial Cell Line, MCF-10 Isolation and Characterization of a Spontaneously Immortalized , 2006 .

[3]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[4]  Brian A. Nosek,et al.  An open investigation of the reproducibility of cancer biology research , 2014, eLife.

[5]  Joshua M. Stuart,et al.  Subtype and pathway specific responses to anticancer compounds in breast cancer , 2011, Proceedings of the National Academy of Sciences.

[6]  Jayanta Debnath,et al.  Morphogenesis and oncogenesis of MCF-10A mammary epithelial acini grown in three-dimensional basement membrane cultures. , 2003, Methods.

[7]  Weihong Xu,et al.  Continuous in vitro exposure to low-dose genistein induces genomic instability in breast epithelial cells. , 2008, Cancer genetics and cytogenetics.

[8]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[9]  B. Cravatt,et al.  Chemical Biology Meets Biological Chemistry Minireview Series* , 2010, The Journal of Biological Chemistry.

[10]  Scott E. Martin,et al.  Reproducible pharmacogenomic profiling of cancer cell line panels , 2016, Nature.

[11]  P. Sorger,et al.  Growth rate inhibition metrics correct for confounders in measuring sensitivity to cancer drugs , 2016, Nature Methods.

[12]  Anne E Carpenter,et al.  CellProfiler: image analysis software for identifying and quantifying cell phenotypes , 2006, Genome Biology.

[13]  P. Gonzalez-Alegre,et al.  Towards precision medicine , 2017 .

[14]  Gordon B Mills,et al.  Inhibition of PI3K/mTOR leads to adaptive resistance in matrix-attached cancer cells. , 2012, Cancer cell.

[15]  Oliver Pelz,et al.  web cellHTS2: A web-application for the analysis of high-throughput screening data , 2010, BMC Bioinformatics.

[16]  W. Kibbe,et al.  Review of Current Methods, Applications, and Data Management for the Bioinformatics Analysis of Whole Exome Sequencing , 2014, Cancer informatics.

[17]  P. Härkönen,et al.  Morphological studies on the effect of taxol on cultured human prostatic cancer cells , 1987, The Prostate.

[18]  Markus List,et al.  Using Docker Compose for the Simple Deployment of an Integrated Drug Target Screening Platform , 2017, J. Integr. Bioinform..

[19]  David R McMillen,et al.  Automated inference procedure for the determination of cell growth parameters. , 2016, Physical review. E.

[20]  N. Marella,et al.  Cytogenetic and cDNA microarray expression analysis of MCF10 human breast cancer progression cell lines. , 2009, Cancer research.

[21]  N. Nowak,et al.  Molecular characterization of the t(3;9) associated with immortalization in the MCF10A cell line. , 2005, Cancer genetics and cytogenetics.

[22]  Benjamin Haibe-Kains,et al.  Inconsistency in large pharmacogenomic studies , 2013, Nature.

[23]  Adam A. Margolin,et al.  The Cancer Cell Line Encyclopedia enables predictive modeling of anticancer drug sensitivity , 2012, Nature.

[24]  Carlos F. Lopez,et al.  An unbiased metric of antiproliferative drug effect in vitro , 2016, Nature Methods.

[25]  Brian A. Nosek,et al.  Making sense of replications , 2017, eLife.

[26]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[27]  Green Dp,et al.  Advances in carpal bone injury and disease. , 1989 .

[28]  Laura M. Heiser,et al.  Tumor-Derived Cell Lines as Molecular Models of Cancer Pharmacogenomics , 2015, Molecular Cancer Research.

[29]  Joshua M. Dempster,et al.  Genetic and transcriptional evolution alters cancer cell line drug response , 2018, Nature.

[30]  Robert Nadon,et al.  Identification and correction of spatial bias are essential for obtaining quality data in high-throughput screening technologies , 2017, Scientific Reports.

[31]  Wolfgang Huber,et al.  Analysis of cell-based RNAi screens , 2006, Genome Biology.

[32]  Adam A. Margolin,et al.  Enabling transparent and collaborative computational analysis of 12 tumor types within The Cancer Genome Atlas , 2013, Nature Genetics.

[33]  P. Clemons,et al.  Target identification and mechanism of action in chemical biology and drug discovery. , 2013, Nature chemical biology.

[34]  J. Arrowsmith.  Trial watch: Phase II failures: 2008–2010 , 2011, Nature Reviews Drug Discovery.

[35]  Andrea Califano,et al.  Detection and removal of spatial bias in multiwell assays , 2016, Bioinform..

[36]  Euan A Ashley,et al.  Performance comparison of whole-genome sequencing platforms , 2011, Nature Biotechnology.

[37]  Levi A Garraway,et al.  Adaptive resistance of melanoma cells to RAF inhibition via reversible induction of a slowly dividing de‐differentiated state , 2017, Molecular systems biology.

[38]  Susanne Heynen-Genel,et al.  Hybrid median filter background estimator for correcting distortions in microtiter plate data. , 2010, Assay and drug development technologies.

[39]  Asserewou Etekpo,et al.  Effects of Metformin and a Mammalian Target of Rapamycin (mTOR) ATP-Competitive Inhibitor on Targeted Metabolomics in Pancreatic Cancer Cell Line. , 2016, Metabolomics : open access.

[40]  S. Ravera,et al.  Metformin Impairs Glucose Consumption and Survival in Calu-1 Cells by Direct Inhibition of Hexokinase-II , 2013, Scientific Reports.

[41]  Marc Hafner,et al.  Designing Drug‐Response Experiments and Quantifying their Results , 2017, Current protocols in chemical biology.

[42]  J. Fletcher,et al.  ABC transporters in cancer: more than just drug efflux pumps , 2010, Nature Reviews Cancer.

[43]  N. Tolliday.  High‐Throughput Assessment of Mammalian Cell Viability by Determination of Adenosine Triphosphate Levels , 2010, Current protocols in chemical biology.

[44]  P. Jänne,et al.  Autocrine Production of Amphiregulin Predicts Sensitivity to Both Gefitinib and Cetuximab in EGFR Wild-type Cancers , 2008, Clinical Cancer Research.

[45]  D. Green,et al.  Advances in carpal bone injury and disease. , 1989, Hand Clinics.

[46]  A. Nekrutenko,et al.  Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences , 2010, Genome Biology.

[47]  P. Sorger,et al.  Reproducibility will only come with data liberation , 2016, Science Translational Medicine.

[48]  S. Ramaswamy,et al.  Systematic identification of genomic markers of drug sensitivity in cancer cells , 2012, Nature.

[49]  Bruce A. Posner,et al.  Diverse drug-resistance mechanisms can emerge from drug-tolerant cancer persister cells , 2016, Nature Communications.

[50]  J. Ioannidis.  Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research. , 2017, JAMA.

[51]  Ralph Weissleder,et al.  Analysis of mitosis and antimitotic drug responses in tumors by in vivo microscopy and single-cell pharmacodynamics. , 2011, Cancer research.

[52]  Marc Hafner,et al.  Measuring Cancer Drug Sensitivity and Resistance in Cultured Cells , 2017, Current protocols in chemical biology.

[53]  Marc Hafner,et al.  Alternative drug sensitivity metrics improve preclinical cancer pharmacogenomics , 2017, Nature Biotechnology.

[54]  Michael P. Morrissey,et al.  Pharmacogenomic agreement between two cancer cell line data sets , 2015, Nature.

[55]  Marc R. Birtwistle,et al.  Drug response consistency in CCLE and CGP , 2016, Nature.

[56]  R. Harris,et al.  Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions , 2017 .

[57]  Jane Fridlyand,et al.  Widespread potential for growth-factor-driven resistance to anticancer kinase inhibitors , 2012, Nature.

[58]  Monya Baker,et al.  Biotech giant publishes failures to confirm high-profile science , 2016, Nature.

[59]  Joshua A. Bittker,et al.  Harnessing Connectivity in a Large-Scale Small-Molecule Sensitivity Dataset. , 2015, Cancer discovery.

[60]  S. Morrison,et al.  Time to do something about reproducibility , 2014, eLife.