Posterior consistency in conditional distribution estimation

A wide variety of priors have been proposed for nonparametric Bayesian estimation of conditional distributions, and there is a clear need for theorems providing conditions on the prior for large support, as well as posterior consistency. Estimation of an uncountable collection of conditional distributions across different regions of the predictor space is a challenging problem, which differs in some important ways from density and mean regression estimation problems. Defining various topologies on the space of conditional distributions, we provide sufficient conditions for posterior consistency focusing on a broad class of priors formulated as predictor-dependent mixtures of Gaussian kernels. This theory is illustrated by showing that the conditions are satisfied for a class of generalized stick-breaking process mixtures in which the stick-breaking lengths are monotone, differentiable functions of a continuous stochastic process. We also provide a set of sufficient conditions for the case where stick-breaking lengths are predictor independent, such as those arising from a fixed Dirichlet process prior.

[1]  Stephen G. Walker,et al.  Slice sampling mixture models , 2011, Stat. Comput..

[2]  S. Ghosal,et al.  Posterior consistency of Gaussian process prior for nonparametric binary regression , 2006, math/0702686.

[3]  G. Roberts,et al.  Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models , 2007, 0710.4228.

[4]  A. W. Vaart,et al.  Reproducing kernel Hilbert spaces of Gaussian priors , 2008, 0805.3252.

[5]  J H Albert,et al.  Sequential Ordinal Modeling with Applications to Survival Data , 2001, Biometrics.

[6]  Jayanta K. Ghosh,et al.  Bayesian density regression with logistic Gaussian process and subspace projection , 2010 .

[7]  Andrea Ongaro,et al.  Discrete random probability measures: a general framework for nonparametric Bayesian inference☆ , 2004 .

[8]  Albert Y. Lo,et al.  On a Class of Bayesian Nonparametric Estimates: I. Density Estimates , 1984 .

[9]  S. MacEachern,et al.  Bayesian Nonparametric Spatial Modeling With Dirichlet Process Mixing , 2005 .

[10]  T. Ferguson.  Prior Distributions on Spaces of Probability Measures , 1974 .

[11]  Ray-Bing Chen,et al.  c-optimal designs for weighted polynomial models , 2005 .

[12]  Jianqing Fan,et al.  Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems , 1996 .

[13]  Catia Scricciolo,et al.  Posterior rates of convergence for Dirichlet mixtures of exponential power densities , 2011 .

[14]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[15]  Alessio Sancetta,et al.  Universality of Bayesian Predictions , 2007 .

[16]  P. Müller,et al.  Bayesian curve fitting using multivariate normal mixtures , 1996 .

[17]  Andriy Norets,et al.  POSTERIOR CONSISTENCY IN CONDITIONAL DENSITY ESTIMATION BY COVARIATE DEPENDENT MIXTURES , 2011, Econometric Theory.

[18]  Fernando A. Quintana,et al.  On the Support of MacEachern’s Dependent Dirichlet Processes and Extensions , 2012 .

[19]  Grace L. Yang,et al.  On Bayes Procedures , 1990 .

[20]  Van Der Vaart,et al.  Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth , 2009, 0908.3556.

[21]  Andriy Norets,et al.  Approximation of conditional densities by smooth mixtures of regressions , 2010, 1010.0581.

[22]  S. Ghosal,et al.  Posterior consistency of Dirichlet mixtures for estimating a transition density , 2007 .

[23]  A. V. D. Vaart,et al.  Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities , 2001 .

[24]  D. Freedman.  On the Bernstein-von Mises Theorem with Infinite Dimensional Parameters , 1999 .

[25]  David B Dunson,et al.  Nonparametric Bayesian models through probit stick-breaking processes. , 2011, Bayesian analysis.

[26]  Yuefeng Wu,et al.  The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation , 2010, J. Multivar. Anal..

[27]  Wesley O Johnson,et al.  Bayesian Nonparametric Nonproportional Hazards Survival Modeling , 2009, Biometrics.

[28]  O. Papaspiliopoulos.  A note on posterior sampling from Dirichlet mixture models , 2008 .

[29]  Debdeep Pati,et al.  Bayesian nonparametric regression with varying residual density , 2014, Annals of the Institute of Statistical Mathematics.

[30]  L. Wasserman,et al.  The consistency of posterior distributions in nonparametric problems , 1999 .

[31]  D. Dunson,et al.  BAYESIAN GENERALIZED PRODUCT PARTITION MODEL , 2010 .

[32]  Geoffrey J. McLachlan,et al.  Multivariate Normal Mixtures , 2005 .

[33]  M. Steel,et al.  Bayesian nonparametric modelling with the Dirichlet process regression smoother , 2010 .

[34]  Steen Pedersen,et al.  Introduction to Continuity , 2015 .

[35]  Taeryon Choi,et al.  Asymptotic properties of posterior distributions in nonparametric regression with non-Gaussian errors , 2009 .

[36]  J. Ghosh,et al.  Posterior consistency of logistic Gaussian process priors in density estimation , 2007 .

[37]  A. Norets,et al.  Bayesian modeling of joint and conditional distributions , 2012 .

[38]  Maria De Iorio,et al.  Bayesian semiparametric inference for multivariate doubly-interval-censored data , 2010, 1101.1415.

[39]  Geoffrey J. McLachlan,et al.  Mixtures of Factor Analyzers , 2000, International Conference on Machine Learning.

[40]  S. Ghosal,et al.  Adaptive Bayesian multivariate density estimation with Dirichlet mixtures , 2011, 1109.6406.

[41]  L. Schwartz.  On Bayes procedures , 1965 .

[42]  A. V. D. Vaart,et al.  Adaptive Bayesian density estimation with location-scale mixtures , 2010 .

[43]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[44]  M. Schervish,et al.  On posterior consistency in nonparametric regression problems , 2007 .

[45]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[46]  J. Rousseau,et al.  BERNSTEIN-VON MISES THEOREM FOR LINEAR FUNCTIONALS OF THE DENSITY , 2009, 0908.4167.

[47]  Joseph B. Kadane,et al.  Error analysis for small angle neutron scattering datasets using Bayesian inference , 2010 .

[48]  N. Pillai,et al.  Bayesian density regression , 2007 .

[49]  Richard Nickl,et al.  Nonparametric Bernstein-von Mises Theorems , 2012 .

[50]  L. Wasserman,et al.  Asymptotic inference for mixture models by using data‐dependent priors , 2000 .

[51]  Yongqiang Tang,et al.  A consistent nonparametric Bayesian procedure for estimating autoregressive conditional densities , 2007, Comput. Stat. Data Anal..

[52]  D. Dunson,et al.  Kernel stick-breaking processes. , 2008, Biometrika.

[53]  Surya T. Tokdar,et al.  Adaptive Convergence Rates of a Dirichlet Process Mixture of Multivariate Normals , 2011, 1111.4148.

[54]  David B. Dunson,et al.  Strong consistency of nonparametric Bayes density estimation on compact metric spaces with applications to specific manifolds , 2012, Annals of the Institute of Statistical Mathematics.

[55]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[56]  Babak Shahbaba,et al.  Nonlinear Models Using Dirichlet Process Mixtures , 2007, J. Mach. Learn. Res..

[57]  Surya T. Tokdar,et al.  Dimension adaptability of Gaussian process models with variable selection and projection , 2011, 1112.0716.

[58]  S. Ghosal,et al.  Kullback Leibler property of kernel mixture priors in Bayesian density estimation , 2007, 0710.2746.

[59]  N. Hjort,et al.  On Bayesian consistency , 2001 .

[60]  R. Adler.  An introduction to continuity, extrema, and related topics for general Gaussian processes , 1990 .

[61]  J. Ghosh,et al.  Posterior consistency for semi-parametric regression problems , 2003 .

[62]  Radford M. Neal,et al.  A Split-Merge Markov chain Monte Carlo Procedure for the Dirichlet Process Mixture Model , 2004 .

[63]  S. MacEachern,et al.  An ANOVA Model for Dependent Random Measures , 2004 .

[64]  D. Dunson,et al.  Nonparametric Bayes Conditional Distribution Modeling With Variable Selection , 2009, Journal of the American Statistical Association.

[65]  J. E. Griffin,et al.  Order-Based Dependent Dirichlet Processes , 2006 .

[66]  J. Ghosh,et al.  POSTERIOR CONSISTENCY OF DIRICHLET MIXTURES IN DENSITY ESTIMATION , 1999 .

[67]  Taeryon Choi,et al.  Alternative posterior consistency results in nonparametric binary regression using Gaussian process priors , 2007 .

[68]  A. V. D. Vaart,et al.  Posterior convergence rates of Dirichlet mixtures at smooth densities , 2007, 0708.1885.