POSTERIOR CONSISTENCY IN CONDITIONAL DENSITY ESTIMATION BY COVARIATE DEPENDENT MIXTURES

This paper considers Bayesian nonparametric estimation of conditional densities by countable mixtures of location-scale densities with covariate dependent mixing probabilities. The mixing probabilities are modeled in two ways. First, we consider finite covariate dependent mixture models, in which the mixing probabilities are proportional to a product of a constant and a kernel and a prior on the number of mixture components is specified. Second, we consider kernel stick-breaking processes for modeling the mixing probabilities. We show that the posterior in these two models is weakly and strongly consistent for a large class of data-generating processes. A simulation study conducted in the paper demonstrates that the models can perform well in small samples.

[1]  S. Walker.  New approaches to Bayesian consistency , 2004, math/0503672.

[2]  Robert Kohn,et al.  Flexible Modeling of Conditional Distributions Using Smooth Mixtures of Asymmetric Student T Densities , 2009 .

[3]  J. Rousseau.  Rates of convergence for the posterior distributions of mixtures of betas and adaptive nonparamatric estimation of the density , 2010, 1001.1615.

[4]  Michael I. Jordan,et al.  Convergence results for the EM approach to mixtures of experts architectures , 1995, Neural Networks.

[5]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[6]  S. Walker.  Invited comment on the paper "Slice Sampling" by Radford Neal , 2003 .

[7]  L. Wasserman,et al.  RATES OF CONVERGENCE FOR THE GAUSSIAN MIXTURE SIEVE , 2000 .

[8]  Hidehiko Ichimura,et al.  Implementing Nonparametric and Semiparametric Estimators , 2006 .

[9]  G. Roberts,et al.  Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models , 2007, 0710.4228.

[10]  Sally Wood,et al.  Bayesian mixture of splines for spatially adaptive nonparametric regression , 2002 .

[11]  Jayanta K. Ghosh,et al.  Bayesian density regression with logistic Gaussian process and subspace projection , 2010 .

[12]  A. Gelfand,et al.  ANALYSIS OF MINNESOTA COLON AND RECTUM CANCER POINT PATTERNS WITH SPATIAL AND NONSPATIAL COVARIATE INFORMATION. , 2009, The annals of applied statistics.

[13]  Rodney C. Wolff,et al.  Methods for estimating a conditional distribution function , 1999 .

[14]  S. Tokdar.  Towards a Faster Implementation of Density Estimation With Logistic Gaussian Process Priors , 2007 .

[15]  M. Escobar.  Estimating Normal Means with a Dirichlet Process Prior , 1994 .

[16]  L. Wasserman,et al.  Practical Bayesian Density Estimation Using Mixtures of Normals , 1997 .

[17]  David J. Nott,et al.  Generalized smooth finite mixtures , 2012 .

[18]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[19]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[20]  Fengchun Peng,et al.  Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-Experts Models With an Applic , 1996 .

[21]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[22]  Eric R. Ziegel,et al.  Practical Nonparametric and Semiparametric Bayesian Statistics , 1998, Technometrics.

[23]  Radford M. Neal.  Slice Sampling , 2003, The Annals of Statistics.

[24]  J. Geweke,et al.  Smoothly mixing regressions , 2007 .

[25]  Van Der Vaart,et al.  Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth , 2009, 0908.3556.

[26]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[27]  A. Norets,et al.  Bayesian modeling of joint and conditional distributions , 2012 .

[28]  Jeffrey S. Racine,et al.  Nonparametric Econometrics: The np Package , 2008 .

[29]  Donald B. Rubin,et al.  Validation of Software for Bayesian Models Using Posterior Quantiles , 2006 .

[30]  Van Der Vaart,et al.  Rates of contraction of posterior distributions based on Gaussian process priors , 2008 .

[31]  R. Kohn,et al.  Simultaneous variable selection and component selection for regression density estimation with mixtures of heteroscedastic experts , 2012 .

[32]  L. Schwartz.  On Bayes procedures , 1965 .

[33]  P. Deb.  Finite Mixture Models , 2008 .

[34]  Andrew R. Barron,et al.  Mixture Density Estimation , 1999, NIPS.

[35]  John Geweke,et al.  Contemporary Bayesian Econometrics and Statistics: Geweke/Contemporary Bayesian Econometrics and Statistics , 2005 .

[36]  Ron Meir,et al.  Density Estimation Through Convex Combinations of Densities: Approximation and Estimation Bounds , 1997, Neural Networks.

[37]  P. Billingsley,et al.  Convergence of Probability Measures , 1969 .

[38]  Jianqing Fan.  Rejoinder: A selective overview of nonparametric methods in financial econometrics , 2004, math/0411034.

[39]  P. Müller,et al.  Bayesian curve fitting using multivariate normal mixtures , 1996 .

[40]  Andriy Norets,et al.  Approximation of conditional densities by smooth mixtures of regressions , 2010, 1010.0581.

[41]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[42]  David B. Dunson,et al.  Posterior consistency in conditional distribution estimation , 2013, J. Multivar. Anal..

[43]  M. V. Jambunathan.  Some Properties of Beta and Gamma Distributions , 1954 .

[44]  Yuefeng Wu,et al.  The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation , 2010, J. Multivar. Anal..

[45]  J. Ghosh,et al.  Posterior consistency of logistic Gaussian process priors in density estimation , 2007 .

[46]  L. Wasserman,et al.  The consistency of posterior distributions in nonparametric problems , 1999 .

[47]  D. Dunson,et al.  Nonparametric Bayes Conditional Distribution Modeling With Variable Selection , 2009, Journal of the American Statistical Association.

[48]  Yves Croissant,et al.  Panel data econometrics in R: The plm package , 2008 .

[49]  Petra E. Todd,et al.  Chapter 74 Implementing Nonparametric and Semiparametric Estimators , 2007 .

[50]  J. Geweke,et al.  Getting It Right , 2004 .

[51]  J. E. Griffin,et al.  Order-Based Dependent Dirichlet Processes , 2006 .

[52]  A. Yatchew,et al.  Nonparametric Regression Techniques in Economics , 1998 .

[53]  J. Ghosh,et al.  POSTERIOR CONSISTENCY OF DIRICHLET MIXTURES IN DENSITY ESTIMATION , 1999 .

[54]  R. Kohn,et al.  Regression Density Estimation Using Smooth Adaptive Gaussian Mixtures , 2007 .

[55]  A. V. D. Vaart,et al.  Adaptive Bayesian density estimation with location-scale mixtures , 2010 .

[56]  S. MacEachern,et al.  An ANOVA Model for Dependent Random Measures , 2004 .

[57]  A. V. D. Vaart,et al.  Convergence rates of posterior distributions , 2000 .

[58]  Justin L. Tobias,et al.  Nonparametric Density and Regression Estimation , 2001 .

[59]  Xiaohong Chen.  Chapter 76 Large Sample Sieve Estimation of Semi-Nonparametric Models , 2007 .

[60]  Jeffrey S. Racine,et al.  Cross-Validation and the Estimation of Conditional Probability Densities , 2004 .

[61]  A. Kottas,et al.  A Bayesian Nonparametric Approach to Inference for Quantile Regression , 2010 .

[62]  O. Papaspiliopoulos.  A note on posterior sampling from Dirichlet mixture models , 2008 .

[63]  J. Geweke,et al.  Contemporary Bayesian Econometrics and Statistics , 2005 .

[64]  D. Dunson,et al.  Kernel stick-breaking processes. , 2008, Biometrika.

[65]  N. Pillai,et al.  Bayesian density regression , 2007 .

[66]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[67]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.