Estimating classification probabilities in high-dimensional diagnostic studies

MOTIVATION Classification algorithms for high-dimensional biological data like gene expression profiles or metabolomic fingerprints are typically evaluated by the number of misclassifications across a test dataset. However, to judge the classification of a single case in the context of clinical diagnosis, we need to assess the uncertainties associated with that individual case rather than the average accuracy across many cases. Reliability of individual classifications can be expressed in terms of class probabilities. While classification algorithms are a well-developed area of research, the estimation of class probabilities is considerably less progressed in biology, with only a few classification algorithms that provide estimated class probabilities. RESULTS We compared several probability estimators in the context of classification of metabolomics profiles. Evaluation criteria included sparseness biases, calibration of the estimator, the variance of the estimator and its performance in identifying highly reliable classifications. We observed that several of them display artifacts that compromise their use in practice. Classification probabilities based on a combination of local cross-validation error rates and monotone regression prove superior in metabolomic profiling. AVAILABILITY The source code written in R is freely available at http://compdiag.uni-regensburg.de/software/probEstimation.shtml. CONTACT inka.appel@klinik.uni-regensburg.de.

[1]  Alexander J. Smola,et al.  Advances in Large Margin Classifiers , 2000 .

[2]  Wolfram Gronwald,et al.  Detection of autosomal dominant polycystic kidney disease by NMR spectroscopic fingerprinting of urine. , 2011, Kidney international.

[3]  C. Sotiriou,et al.  Taking gene-expression profiling to the clinic: when will molecular signatures become relevant to patient care? , 2007, Nature Reviews Cancer.

[4]  Cor J. Veenman,et al.  A protocol for building and evaluating predictors of disease state based on microarray data , 2005, Bioinform..

[5]  Bianca Zadrozny,et al.  Transforming classifier scores into accurate multiclass probability estimates , 2002, KDD.

[6]  S. Dudoit,et al.  Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data , 2002 .

[7]  Maqc Consortium.  The MicroArray Quality Control ( MAQC )-II study of common practices for the development and validation of microarray-based predictive models , 2012 .

[8]  R. Tibshirani,et al.  Diagnosis of multiple cancer types by shrunken centroids of gene expression , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[9]  A. Dawid.  The Well-Calibrated Bayesian , 1982 .

[10]  Kevin C. Dorff,et al.  The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models , 2010, Nature Biotechnology.

[11]  H. D. Brunk,et al.  AN EMPIRICAL DISTRIBUTION FUNCTION FOR SAMPLING WITH INCOMPLETE INFORMATION , 1955 .

[12]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[13]  Satoru Miyano,et al.  Predicting gene regulation by sigma factors in Bacillus subtilis from genome-wide data , 2004, ISMB/ECCB.

[14]  Rich Caruana,et al.  Predicting good probabilities with supervised learning , 2005, ICML.

[15]  Manolis Tsiknakis,et al.  Maturation of a central , 1996 .

[16]  Weida Tong,et al.  DNA Microarrays Are Predictive of Cancer Prognosis: A Re-evaluation , 2010, Clinical Cancer Research.

[17]  C Hill,et al.  Interpretation of microarray data in cancer , 2007, British Journal of Cancer.

[18]  R. Spang,et al.  Predicting the clinical status of human breast cancer by using gene expression profiles , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[19]  Adrian Wiestner,et al.  A gene expression-based method to diagnose clinically distinct subgroups of diffuse large B cell lymphoma , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Mark R. Viant,et al.  Improved classification accuracy in 1- and 2-dimensional NMR metabolomics data using the variance stabilising generalised logarithm transformation , 2007, BMC Bioinformatics.

[21]  Geoffrey J McLachlan,et al.  Selection bias in gene extraction on the basis of microarray gene-expression data , 2002, Proceedings of the National Academy of Sciences of the United States of America.