Efficient Bayesian Multivariate Surface Regression

Methods for choosing a fixed set of knot locations in additive spline models are fairly well established in the statistical literature. The curse of dimensionality makes it nontrivial to extend these methods to nonadditive surface models, especially when there are more than a couple of covariates. We propose a multivariate Gaussian surface regression model that combines both additive splines and interactive splines, and a highly efficient Markov chain Monte Carlo algorithm that updates all the knot locations jointly. We use shrinkage prior to avoid overfitting with different estimated shrinkage factors for the additive and surface part of the model, and also different shrinkage parameters for the different response variables. Simulated data and an application to firm leverage data show that the approach is computationally efficient, and that allowing for freely estimated knot locations can offer a substantial improvement in out-of-sample predictive performance.

[1]  R. Rajan,et al.  What Do We Know About Capital Structure? Some Evidence from International Data , 1994 .

[2]  R. Kohn,et al.  Nonparametric regression using Bayesian variable selection , 1996 .

[3]  Martin D. Buhmann,et al.  Radial Basis Functions: Theory and Implementations: Preface , 2003 .

[4]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[5]  H. Müller,et al.  Kernel estimation of regression functions , 1979 .

[6]  R. Kass.  Bayes Factors in Practice , 1993 .

[7]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[8]  H. Chipman,et al.  BART: Bayesian Additive Regression Trees , 2008, 0806.3286.

[9]  Arto Luoma,et al.  Bayesian Model Selection , 2016 .

[10]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[11]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[12]  C. Andrieu,et al.  Bayesian model selection and parameter estimation of nuclear emission spectra using RJMCMC , 2003 .

[13]  J. Geweke,et al.  Optimal Prediction Pools , 2008 .

[14]  Adrian F. M. Smith,et al.  Automatic Bayesian curve fitting , 1998 .

[15]  A. Zellner.  An Introduction to Bayesian Inference in Econometrics , 1971 .

[16]  John Geweke,et al.  Interpretation and inference in mixture models: Simple MCMC works , 2007, Comput. Stat. Data Anal..

[17]  E. Nadaraya.  On Estimating Regression , 1964 .

[18]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[19]  H. Luetkepohl.  The Handbook of Matrices , 1996 .

[20]  David S. Leslie,et al.  A general approach to heteroscedastic linear regression , 2007, Stat. Comput..

[21]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[22]  B. Mallick,et al.  Generalized Nonlinear Modeling With Multivariate Free-Knot Regression Splines , 2003 .

[23]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[24]  C. R. Rao,et al.  Solutions to some functional equations and the applications , 1968 .

[25]  David J Nott,et al.  Sampling Schemes for Bayesian Variable Selection in Generalized Linear Models , 2004 .

[26]  Dani Gamerman,et al.  Sampling from the posterior distribution in generalized linear mixed models , 1997, Stat. Comput..

[27]  David J. Nott,et al.  Generalized smooth finite mixtures , 2012 .

[28]  R. Tibshirani,et al.  Generalized Additive Models , 1991 .

[29]  Refik Soyer,et al.  Bayesian Methods for Nonlinear Classification and Regression , 2004, Technometrics.

[30]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[31]  Sally Wood,et al.  Bayesian mixture of splines for spatially adaptive nonparametric regression , 2002 .

[32]  R. Kass,et al.  Bayesian curve-fitting with free-knot splines , 2001 .

[33]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[34]  John Geweke,et al.  Evaluating the accuracy of sampling-based approaches to the calculation of posterior moments , 1991 .

[35]  B. Ripley,et al.  Semiparametric Regression: Preface , 2003 .