Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part II: Consistency and Performance Analysis

In Part I of a two part study on the MSE performance of Bayesian error estimation, we have derived analytical expressions for MSE conditioned on the sample for Bayesian error estimators and arbitrary error estimators in two Bayesian models: discrete classification with Dirichlet priors and linear classification of Gaussian distributions with normal-inverse-Wishart priors. Here, in Part II, we examine the consistency of Bayesian error estimation and provide several simulation studies that illustrate the concept of conditional MSE and how it may be used in practice. A salient application is censored sampling, where sample points are collected one at a time until the conditional MSE reaches a stopping criterion.

[1]  Mark E. Johnson,et al.  Multivariate Statistical Simulation , 1989, International Encyclopedia of Statistical Science.

[2]  Edward R. Dougherty,et al.  Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part I: Definition and the Bayesian MMSE Error Estimator for Discrete Classification , 2011, IEEE Transactions on Signal Processing.

[3]  Mark E. Johnson Multivariate Statistical Simulation: Johnson/Multivariate , 1987 .

[4]  Ulisses Braga-Neto,et al.  On the sampling distribution of resubstitution and leave-one-out error estimators for linear classifiers , 2009, Pattern Recognit..

[5]  D. Freedman On the Asymptotic Behavior of Bayes' Estimates in the Discrete Case , 1963 .

[6]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[7]  A. Brix Bayesian Data Analysis, 2nd edn , 2005 .

[8]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[9]  Edward R. Dougherty,et al.  Application of the Bayesian MMSE estimator for classification error to gene expression microarray data , 2011, Bioinform..

[10]  Van,et al.  A gene-expression signature as a predictor of survival in breast cancer. , 2002, The New England journal of medicine.

[11]  Edward R. Dougherty,et al.  Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part II: Linear Classification of Gaussian Models , 2011, IEEE Transactions on Signal Processing.

[12]  Yudong D. He,et al.  A Gene-Expression Signature as a Predictor of Survival in Breast Cancer , 2002 .

[13]  Edward R. Dougherty,et al.  Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part I: Representation , 2012, IEEE Transactions on Signal Processing.

[14]  D. Freedman,et al.  On the consistency of Bayes estimates , 1986 .