Bayesian nonparametric regression with varying residual density

We consider the problem of robust Bayesian inference on the mean regression function allowing the residual density to change flexibly with predictors. The proposed class of models is based on a Gaussian process (GP) prior for the mean regression function and mixtures of Gaussians for the collection of residual densities indexed by predictors. Initially considering the homoscedastic case, we propose priors for the residual density based on probit stick-breaking mixtures. We provide sufficient conditions to ensure strong posterior consistency in estimating the regression function, generalizing existing theory focused on parametric residual distributions. The homoscedastic priors are generalized to allow residual densities to change nonparametrically with predictors through incorporating GP in the stick-breaking components. This leads to a robust Bayesian regression procedure that automatically down-weights outliers and influential observations in a locally adaptive manner. The methods are illustrated using simulated and real data applications.

[1]  R. Weiss.  An approach to Bayesian sensitivity analysis , 1996 .

[2]  J. E. Griffin,et al.  Order-Based Dependent Dirichlet Processes , 2006 .

[3]  Van Der Vaart,et al.  Adaptive Bayesian estimation using a Gaussian random field with inverse Gamma bandwidth , 2009, 0908.3556.

[4]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[5]  Andrea Ongaro,et al.  Discrete random probability measures: a general framework for nonparametric Bayesian inference☆ , 2004 .

[6]  D. Dunson,et al.  Nonparametric Bayes Conditional Distribution Modeling With Variable Selection , 2009, Journal of the American Statistical Association.

[7]  R. Adler.  An introduction to continuity, extrema, and related topics for general Gaussian processes , 1990 .

[8]  A. P. Dawid,et al.  Regression and Classification Using Gaussian Process Priors , 2009 .

[9]  R. Arellano-Valle,et al.  Bayesian sensitivity analysis in elliptical linear regression models , 2000 .

[10]  Posterior Consistency for some Semi-parametric Problems , 2008 .

[11]  David B Dunson,et al.  Nonparametric Bayesian models through probit stick-breaking processes. , 2011, Bayesian analysis.

[12]  S. Chib,et al.  Additive cubic spline regression with Dirichlet process mixture errors , 2010 .

[13]  O. Papaspiliopoulos.  A note on posterior sampling from Dirichlet mixture models , 2008 .

[14]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[15]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[16]  David B. Dunson,et al.  Posterior consistency in conditional distribution estimation , 2013, J. Multivar. Anal..

[17]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[18]  M. Escobar,et al.  Bayesian Density Estimation and Inference Using Mixtures , 1995 .

[19]  Stephen G. Walker,et al.  Slice sampling mixture models , 2011, Stat. Comput..

[20]  Robert Kohn,et al.  Estimation and variable selection in nonparametric heteroscedastic regression , 2003, Stat. Comput..

[21]  Malcolm R Leadbetter,et al.  Stationary and Related Stochastic Processes: Sample Function Properties and Their Applications , 1967 .

[22]  J. Ghosh,et al.  Posterior consistency for semi-parametric regression problems , 2003 .

[23]  H. Chipman,et al.  Bayesian Additive Regression Trees , 2006 .

[24]  T. Ferguson.  Prior Distributions on Spaces of Probability Measures , 1974 .

[25]  S. Ghosal,et al.  Kullback Leibler property of kernel mixture priors in Bayesian density estimation , 2007, 0710.2746.

[26]  Christopher Holmes,et al.  Bayesian Methods for Nonlinear Classification and Regressing , 2002 .

[27]  D. Burr,et al.  A Bayesian Semiparametric Model for Random-Effects Meta-Analysis , 2005 .

[28]  Hani Doss.  Bayesian Nonparametric Estimation of the Median; Part I: Computation of the Estimates , 1985 .

[29]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[30]  Audris Mockus,et al.  A nonparametric Bayes method for isotonic regression , 1995 .

[31]  G. Roberts,et al.  Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models , 2007, 0710.4228.

[32]  Michael,et al.  On a Class of Bayesian Nonparametric Estimates : I . Density Estimates , 2008 .

[33]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[34]  M. West.  On scale mixtures of normal distributions , 1987 .

[35]  Taeryon Choi,et al.  Asymptotic properties of posterior distributions in nonparametric regression with non-Gaussian errors , 2009 .

[36]  David J. Nott,et al.  Semiparametric estimation of mean and variance functions for non-Gaussian data , 2006, Comput. Stat..

[37]  D. Dunson,et al.  Kernel stick-breaking processes. , 2008, Biometrika.

[38]  harald Cramer,et al.  Stationary And Related Stochastic Processes , 1967 .

[39]  M. Schervish,et al.  Posterior Consistency in Nonparametric Regression Problems under Gaussian Process Priors , 2004 .

[40]  Justinas Pelenis.  Bayesian Semiparametric Regression , 2012 .

[41]  N. Pillai,et al.  Bayesian density regression , 2007 .

[42]  M. Schervish,et al.  On posterior consistency in nonparametric regression problems , 2007 .

[43]  Jeremy MG Taylor,et al.  Robust Statistical Modeling Using the t Distribution , 1989 .

[44]  S. Ghosal,et al.  Posterior consistency of Gaussian process prior for nonparametric binary regression , 2006, math/0702686.

[45]  Jon A. Wellner,et al.  Weak Convergence and Empirical Processes: With Applications to Statistics , 1996 .

[46]  A. Raftery,et al.  How Many Iterations in the Gibbs Sampler , 1991 .

[47]  S. MacEachern,et al.  A semiparametric Bayesian model for randomised block designs , 1996 .

[48]  M. Steel,et al.  Bayesian nonparametric modelling with the Dirichlet process regression smoother , 2010 .

[49]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[50]  Andriy Norets,et al.  POSTERIOR CONSISTENCY IN CONDITIONAL DENSITY ESTIMATION BY COVARIATE DEPENDENT MIXTURES , 2011, Econometric Theory.

[51]  John Geweke,et al.  Evaluating the accuracy of sampling-based approaches to the calculation of posterior moments , 1991 .

[52]  A. W. Vaart,et al.  Reproducing kernel Hilbert spaces of Gaussian priors , 2008, 0805.3252.

[53]  R. Kohn,et al.  Locally Adaptive Semiparametric Estimation of the Mean and Variance Functions in Regression Models , 2006 .

[54]  Refik Soyer,et al.  Bayesian Methods for Nonlinear Classification and Regression , 2004, Technometrics.

[55]  M. West.  Outlier Models and Prior Distributions in Bayesian Linear Regression , 1984 .

[56]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[57]  P. Müller,et al.  Bayesian curve fitting using multivariate normal mixtures , 1996 .

[58]  Helio S. Migon,et al.  Objective Bayesian analysis for the Student-t regression model , 2008 .

[59]  J H Albert,et al.  Sequential Ordinal Modeling with Applications to Survival Data , 2001, Biometrics.

[60]  Lancelot F. James,et al.  Bayesian Inference Via Classes of Normalized Random Measures , 2005, math/0503394.

[61]  L. Schwartz.  On Bayes procedures , 1965 .

[62]  A. Gelfand,et al.  Bayesian Semiparametric Median Regression Modeling , 2001 .

[63]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .