A comparative review of variable selection techniques for covariate dependent Dirichlet process mixture models

Dirichlet Process Mixture (DPM) models have been increasingly employed to specify random partition models that take into account possible patterns within covariates. Furthermore with large numbers of covariates methods for selecting the most important covariates have been proposed. Commonly the covariates are chosen either for their importance in determining the clustering of the observations or for their effect on the level of a response variable (when a regression model is specified). Typically both strategies involve the specification of latent indicators that regulate the inclusion of the covariates in the model. Common examples involve the use of spike and slab prior distributions. In this work we review the most relevant DPM models that include covariate information in the induced partition of the observations and we focus on available variable selection techniques for these models. We highlight the main features of each model and demonstrate them using simulations and a real data application. The Canadian Journal of Statistics 45: 254–273; 2017 © 2017 Statistical Society of Canada

[1]  Peter Müller,et al.  Cluster‐Specific Variable Selection for Product Partition Models , 2015 .

[2]  G. Baio,et al.  Variable selection in covariate dependent random partition models: an application to urinary tract infection , 2015, Statistics in medicine.

[3]  Sonia Petrone,et al.  A Predictive Study of Dirichlet Process Mixture Models for Curve Fitting , 2014, Scandinavian journal of statistics, theory and applications.

[4]  David B. Dunson,et al.  Nonparametric Bayes inference on conditional independence , 2014, 1404.1429.

[5]  K. Mengersen,et al.  Bayesian nonparametric dependent model for partially replicated data: The influence of fuel spills on species diversity , 2014, 1402.3093.

[6]  Sylvia Richardson,et al.  Exploring dependence between categorical variables: Benefits and limitations of using variable selection within Bayesian clustering in relation to log-linear modelling with interaction terms , 2014, Journal of statistical planning and inference.

[7]  Sylvia Richardson,et al.  PReMiuM: An R Package for Profile Regression Mixture Models Using Dirichlet Processes. , 2013, Journal of statistical software.

[8]  P. Müller,et al.  Effect on Prediction When Modeling Covariates in Bayesian Nonparametric Models , 2013, Journal of statistical theory and practice.

[9]  Nicholas J. Foti,et al.  A Survey of Non-Exchangeable Priors for Bayesian Nonparametric Models , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Sylvia Richardson,et al.  Exploring Data From Genetic Association Studies Using Bayesian Variable Selection and the Dirichlet Process: Application to Searching for Gene × Gene Patterns , 2012, Genetic epidemiology.

[11]  David B Dunson,et al.  Nonparametric Bayesian models through probit stick-breaking processes. , 2011, Bayesian analysis.

[12]  David B. Dunson,et al.  Logistic Stick-Breaking Process , 2011, J. Mach. Learn. Res..

[13]  D. Dunson,et al.  The local Dirichlet process , 2011, Annals of the Institute of Statistical Mathematics.

[14]  Peter Müller,et al.  A Product Partition Model With Regression on Covariates , 2011, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[15]  P. Müller,et al.  Random Partition Models with Regression on Covariates. , 2010, Journal of statistical planning and inference.

[16]  M. Steel,et al.  Bayesian nonparametric modelling with the Dirichlet process regression smoother , 2010 .

[17]  D. Dunson,et al.  BAYESIAN GENERALIZED PRODUCT PARTITION MODEL , 2010 .

[18]  Sylvia Richardson,et al.  Bayesian profile regression with an application to the National Survey of Children's Health. , 2010, Biostatistics.

[19]  David B. Dunson,et al.  Bayesian Nonparametrics: Nonparametric Bayes applications to biostatistics , 2010 .

[20]  A. Lijoi,et al.  Models Beyond the Dirichlet Process , 2009 .

[21]  D. Dunson,et al.  Nonparametric Bayes Conditional Distribution Modeling With Variable Selection , 2009, Journal of the American Statistical Association.

[22]  Warren B. Powell,et al.  Dirichlet Process Mixtures of Generalized Linear Models , 2009, J. Mach. Learn. Res..

[23]  Wesley O Johnson,et al.  Bayesian Nonparametric Nonproportional Hazards Survival Modeling , 2009, Biometrics.

[24]  R. O’Hara,et al.  A review of Bayesian variable selection methods: what, how and which , 2009 .

[25]  D. Dunson,et al.  Kernel stick-breaking processes. , 2008, Biometrika.

[26]  Abel Rodriguez,et al.  Bayesian hierarchically weighted finite mixture models for samples of distributions. , 2008, Biostatistics.

[27]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[28]  Jason A. Duan,et al.  Generalized spatial dirichlet process models , 2007 .

[29]  G. Roberts,et al.  Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models , 2007, 0710.4228.

[30]  P. Green,et al.  Bayesian Model-Based Clustering Procedures , 2007 .

[31]  Brian J. Reich,et al.  A MULTIVARIATE SEMIPARAMETRIC BAYESIAN SPATIAL MODELING FRAMEWORK FOR HURRICANE SURFACE WIND FIELDS , 2007, 0709.0427.

[32]  N. Pillai,et al.  Bayesian density regression , 2007 .

[33]  Babak Shahbaba,et al.  Nonlinear Models Using Dirichlet Process Mixtures , 2007, J. Mach. Learn. Res..

[34]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[35]  J. E. Griffin,et al.  Order-Based Dependent Dirichlet Processes , 2006 .

[36]  S. MacEachern,et al.  Bayesian Nonparametric Spatial Modeling With Dirichlet Process Mixing , 2005 .

[37]  S. MacEachern,et al.  An ANOVA Model for Dependent Random Measures , 2004 .

[38]  Christopher M. Bishop,et al.  Bayesian Hierarchical Mixtures of Experts , 2002, UAI.

[39]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[40]  H. Ishwaran,et al.  Markov chain Monte Carlo in approximate Dirichlet and beta two-parameter process hierarchical models , 2000 .

[41]  Radford M. Neal.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[42]  Peter Müller,et al.  A Bayesian Population Model with Hierarchical Mixture Priors Applied to Blood Count Data , 1997 .

[43]  P. Müller,et al.  Bayesian curve fitting using multivariate normal mixtures , 1996 .

[44]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[45]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994, Neural Computation.

[46]  J. Hartigan,et al.  Product Partition Models for Change Point Problems , 1992 .

[47]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[48]  Albert Y. Lo,et al.  On a Class of Bayesian Nonparametric Estimates: I. Density Estimates , 1984 .

[49]  D. Binder.  Bayesian cluster analysis , 1978 .

[50]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[51]  D. Blackwell.  Discreteness of Ferguson Selections , 1973 .

[52]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[53]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[54]  W. Ewens.  The sampling theory of selectively neutral alleles. , 1972, Theoretical population biology.

[55]  D. Blei.  Bayesian Nonparametrics I , 2016 .

[56]  P. McCullagh.  Partition models , 2015 .

[57]  David B. Dunson,et al.  Improving prediction from dirichlet process mixtures via enrichment , 2014, J. Mach. Learn. Res..

[58]  Antonio Lijoi,et al.  Bayesian Nonparametrics: Models beyond the Dirichlet process , 2010 .

[59]  Marina Vannucci,et al.  Spiked Dirichlet Process Prior for Bayesian Multiple Hypothesis Testing in Random Effects Models. , 2009, Bayesian analysis.

[60]  Michael,et al.  On a Class of Bayesian Nonparametric Estimates : I . Density Estimates , 2008 .

[61]  Carlos M. Carvalho,et al.  Sparse Statistical Modelling in Gene Expression Genomics , 2006 .

[62]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[63]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[64]  J. Pitman.  Some developments of the Blackwell-MacQueen urn scheme , 1996 .