Variational Bayes for Hierarchical Mixture Models

In recent years, sparse classification problems have emerged in many fields of study. Finite mixture models have been developed to facilitate Bayesian inference where parameter sparsity is substantial. Classification with finite mixture models is based on the posterior expectation of latent indicator variables. These quantities are typically estimated using the expectation-maximization (EM) algorithm in an empirical Bayes approach or Markov chain Monte Carlo (MCMC) in a fully Bayesian approach. MCMC is limited in applicability where high-dimensional data are involved because its sampling-based nature leads to slow computations and hard-to-monitor convergence. In this chapter, we investigate the feasibility and performance of variational Bayes (VB) approximation in a fully Bayesian framework. We apply the VB approach to fully Bayesian versions of several finite mixture models that have been proposed in bioinformatics, and find that it achieves desirable speed and accuracy in sparse classification with finite mixture models for high-dimensional data.

[1]  D.G. Tzikas,et al.  The variational approximation for Bayesian inference , 2008, IEEE Signal Processing Magazine.

[2]  Adrian Corduneanu,et al.  Variational Bayesian Model Selection for Mixture Distributions , 2001 .

[3]  John T. Ormerod,et al.  Grid based variational approximations , 2011, Comput. Stat. Data Anal..

[4]  Thomas Brendan Murphy,et al.  Variational Bayesian inference for the Latent Position Cluster Model , 2009, NIPS 2009.

[5]  Gordon K Smyth,et al.  Statistical Applications in Genetics and Molecular Biology Linear Models and Empirical Bayes Methods for Assessing Differential Expression in Microarray Experiments , 2011 .

[6]  D. M. Titterington,et al.  Variational approximations in Bayesian model selection for finite mixture distributions , 2007, Comput. Stat. Data Anal..

[7]  Benjamin A. Logsdon,et al.  A variational Bayes algorithm for fast and accurate multiple locus genome-wide association analysis , 2010, BMC Bioinformatics.

[8]  Ronald Christensen,et al.  Bayesian Ideas and Data Analysis: An Introduction for Scientists and Statisticians , 2010 .

[9]  Andrea Bergmann,et al.  Statistical Parametric Mapping The Analysis Of Functional Brain Images , 2016 .

[10]  Guifang Fu,et al.  The Bayesian lasso for genome-wide association studies , 2011, Bioinform..

[11]  Andrew G. Clark,et al.  Mapping Multiple Quantitative Trait Loci by Bayesian Classification , 2005, Genetics.

[12]  Jean-Michel Marin,et al.  Mean-field variational approximate Bayesian inference for latent variable models , 2007, Comput. Stat. Data Anal..

[13]  David J. Spiegelhalter,et al.  Microarrays, Empirical Bayes and the Two-Groups Model. Comment. , 2008 .

[14]  Haiyuan Yu,et al.  A Bayesian Mixture Model for Comparative Spectral Count Data in Shotgun Proteomics , 2011, Molecular & Cellular Proteomics.

[15]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[16]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[17]  Justin Grimmer An Introduction to Bayesian Inference via Variational Approximations , 2011, Political Analysis.

[18]  Gordon K. Smyth,et al.  limma: Linear Models for Microarray Data , 2005 .

[19]  Karl J. Friston,et al.  Statistical parametric mapping , 2013 .

[20]  Martin T. Wells,et al.  Laplace Approximated EM Microarray Analysis: An Empirical Bayes Approach for Comparative Microarray Experiments , 2010, 1101.0905.

[21]  Zitong Li,et al.  Estimation of Quantitative Trait Locus Effects with Epistasis by Variational Bayes Algorithms , 2012, Genetics.

[22]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[23]  Antti Honkela,et al.  Unsupervised Variational Bayesian Learning of Nonlinear Models , 2004, NIPS.

[24]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[25]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[26]  V. Šmídl,et al.  The Variational Bayes Method in Signal Processing , 2005 .

[27]  Jean-Michel Marin,et al.  Bayesian Core: A Practical Approach to Computational Bayesian Statistics , 2010 .

[28]  Nando de Freitas,et al.  Variational MCMC , 2001, UAI.

[29]  Bo Wang,et al.  Inadequacy of interval estimates corresponding to variational Bayesian approximations , 2005, AISTATS.

[30]  Matthew J. Beal Variational algorithms for approximate Bayesian inference , 2003 .

[31]  Charles M. Bishop Variational principal components , 1999 .

[32]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[33]  Luping Zhao,et al.  A Bayesian Semiparametric Temporally-Stratified Proportional Hazards Model with Spatial Frailties. , 2012, Bayesian analysis.

[34]  M. Wand,et al.  Variational Bayesian Inference for Parametric and Nonparametric Regression With Missing Data , 2011 .

[35]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Bradley P. Carlin,et al.  Markov Chain Monte Carlo conver-gence diagnostics: a comparative review , 1996 .

[37]  S. Dudoit,et al.  Microarray expression profiling identifies genes with altered expression in HDL-deficient mice. , 2000, Genome research.

[38]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[39]  Tommi S. Jaakkola,et al.  Tutorial on variational approximation methods , 2000 .

[40]  P. Deb Finite Mixture Models , 2008 .

[41]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[42]  M. Wand,et al.  Mean field variational bayes for elaborate distributions , 2011 .

[43]  David J. Spiegelhalter,et al.  VIBES: A Variational Inference Engine for Bayesian Networks , 2002, NIPS.

[44]  Andrew E. Teschendorff,et al.  A variational Bayesian mixture modelling framework for cluster analysis of gene-expression data , 2005, Bioinform..

[45]  M. Wand,et al.  Explaining Variational Approximations , 2010 .

[46]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[47]  Ciprian Crainiceanu,et al.  Functional regression via variational Bayes. , 2011, Electronic journal of statistics.

[48]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .