A note on statistical repeatability and study design for high‐throughput assays

Characterizing the technical precision of measurements is a necessary stage in the planning of experiments and in the formal sample size calculation for optimal design. Instruments that measure multiple analytes simultaneously, such as in high‐throughput assays arising in biomedical research, pose particular challenges from a statistical perspective. The current most popular method for assessing precision of high‐throughput assays is by scatterplotting data from technical replicates. Here, we question the statistical rationale of this approach from both an empirical and theoretical perspective, illustrating our discussion using four example data sets from different genomic platforms. We demonstrate that such scatterplots convey little statistical information of relevance and are potentially highly misleading. We present an alternative framework for assessing the precision of high‐throughput assays and planning biomedical experiments. Our methods are based on repeatability—a long‐established statistical quantity also known as the intraclass correlation coefficient. We provide guidance and software for estimation and visualization of repeatability of high‐throughput assays, and for its incorporation into study design. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

[1]  A. Beckett,et al.  AKUFO AND IBARAPA. , 1965, Lancet.

[2]  F. Collins,et al.  A new initiative on precision medicine. , 2015, The New England journal of medicine.

[3]  Stephen C. Harris,et al.  Rat toxicogenomic study reveals analytical consistency across microarray platforms , 2006, Nature Biotechnology.

[4]  Torbjörn Falkmer,et al.  The Case for Using the Repeatability Coefficient When Calculating Test–Retest Reliability , 2013, PloS one.

[5]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[6]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[7]  Peter Donnelly,et al.  A Genome-Wide Metabolic QTL Analysis in Europeans Implicates Two Loci Shaped by Recent Positive Selection , 2011, PLoS genetics.

[8]  C. Ball,et al.  Repeatability of published microarray gene expression analyses , 2009, Nature Genetics.

[9]  P. Collins,et al.  Performance comparison of one-color and two-color platforms within the Microarray Quality Control (MAQC) project , 2006, Nature Biotechnology.

[10]  Mattias Rantalainen,et al.  Variance decomposition of protein profiles from antibody arrays using a longitudinal twin model , 2011, Proteome Science.

[11]  Noor Azina Ismail,et al.  Statistical Methods Used to Test for Agreement of Medical Instruments Measuring Continuous Variables in Method Comparison Studies: A Systematic Review , 2012, PloS one.

[12]  Martin Krzywinski,et al.  Points of significance: Power and sample size , 2013, Nature Methods.

[13]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[14]  Jennifer L. Osborn,et al.  Direct multiplexed measurement of gene expression with color-coded probe pairs , 2008, Nature Biotechnology.

[15]  Matthew E. Wolak,et al.  Guidelines for estimating repeatability , 2012 .

[16]  Quin F. Wills,et al.  MicroRNA Expression in Abdominal and Gluteal Adipose Tissue Is Associated with mRNA Expression Levels and Partly Genetically Driven , 2011, PloS one.

[17]  Kevin J Woollard,et al.  A proteomic analysis of C-reactive protein stimulated THP-1 monocytes , 2011, Proteome Science.

[18]  Harvey J Cohen,et al.  An Overview of Variance Inflation Factors for Sample-Size Calculation , 2003, Evaluation & the health professions.

[19]  Shinichi Nakagawa,et al.  Repeatability for Gaussian and non‐Gaussian data: a practical guide for biologists , 2010, Biological reviews of the Cambridge Philosophical Society.

[20]  Laura J. Scott,et al.  Stratifying Type 2 Diabetes Cases by BMI Identifies Genetic Risk Variants in LAMA1 and Enrichment for Risk Variants in Lean Compared to Obese Cases , 2012, PLoS genetics.

[21]  David M. Simcha,et al.  Tackling the widespread and critical impact of batch effects in high-throughput data , 2010, Nature Reviews Genetics.

[22]  S. Tringe,et al.  Validation of two ribosomal RNA removal methods for microbial metatranscriptomics , 2010, Nature Methods.

[23]  John D. Storey.  A direct approach to false discovery rates , 2002 .

[24]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[25]  S. Letovsky,et al.  Quantification of the yeast transcriptome by single-molecule sequencing , 2009, Nature Biotechnology.

[26]  E. Marcotte,et al.  Absolute protein expression profiling estimates the relative contributions of transcriptional and translational regulation , 2007, Nature Biotechnology.

[27]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[28]  B. Williams,et al.  Mapping and quantifying mammalian transcriptomes by RNA-Seq , 2008, Nature Methods.

[29]  J. Ioannidis.  Why Most Published Research Findings Are False , 2019, CHANCE.

[30]  M. Peplow.  The 100 000 Genomes Project , 2016, British Medical Journal.

[31]  G. Monette,et al.  Generalized Collinearity Diagnostics , 1992 .

[32]  Peter J. Bickel,et al.  Measuring reproducibility of high-throughput experiments , 2011, 1110.4705.

[33]  D. Altman,et al.  Measuring agreement in method comparison studies , 1999, Statistical methods in medical research.

[34]  Quin F. Wills,et al.  Coexpression Network Analysis in Abdominal and Gluteal Adipose Tissue Reveals Regulatory Genetic Loci for Metabolic Syndrome and Related Phenotypes , 2012, PLoS genetics.