Multivariate Stochastic Process Models for Correlated Responses of Mixed Type

We propose a new model for correlated outputs of mixed type, such as continuous and binary outputs, with a particular focus on joint regression and classification, motivated by an application in constrained optimization for com- puter simulation modeling. Our framework is based upon multivariate stochastic processes, extending Gaussian process methodology for modeling of continuous multivariate spatial outputs by adding a latent process structure that allows for joint modeling of a variety of types of correlated outputs. In addition, we imple- ment fully Bayesian inference using particle learning, which allows us to conduct fast sequential inference. We demonstrate the effectiveness of our proposed meth- ods on both synthetic examples and a real world hydrology computer experiment optimization problem where it is helpful to model the black box objective function as correlated with satisfaction of the constraint.

[1]  E. Chang,et al.  Time Series Modelling , 2021 .

[2]  L. Vogt.  Statistics For Spatial Data , 2016 .

[3]  David Volent Lindberg,et al.  Optimization Under Constraints by Applying an Asymmetric Entropy Measure , 2015 .

[4]  Zenglin Xu,et al.  Scalable Nonparametric Multiway Data Analysis , 2015, AISTATS.

[5]  Herbert K. H. Lee,et al.  Sequential process convolution Gaussian process models via particle learning , 2014 .

[6]  Yuan Qi,et al.  DinTucker: Scaling up Gaussian process models on multidimensional arrays with billions of elements , 2013, ArXiv.

[7]  Surya T. Tokdar,et al.  Computer emulation with non-stationary Gaussian processes , 2013, 1308.4756.

[8]  Jeremy E. Oakley,et al.  Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures , 2013, Technometrics.

[9]  Zenglin Xu,et al.  Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis , 2011, ICML.

[10]  Hisashi Kashima,et al.  Self-measuring Similarity for Multi-task Gaussian Process , 2011, ICML Unsupervised and Transfer Learning.

[11]  Robert B. Gramacy,et al.  Cases for the nugget in modeling computer experiments , 2010, Statistics and Computing.

[12]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[13]  Nathan M. Urban,et al.  A comparison of Latin hypercube and grid ensemble designs for the multivariate emulation of an Earth system model , 2010, Comput. Geosci..

[14]  A. O'Hagan,et al.  Bayesian emulation of complex multi-output and dynamic computer models , 2010 .

[15]  Nicholas G. Polson,et al.  Particle Learning and Smoothing , 2010, 1011.1098.

[16]  Herbert K. H. Lee,et al.  Bayesian Guided Pattern Search for Robust Local Optimization , 2009, Technometrics.

[17]  Robert B. Gramacy,et al.  Particle Learning of Gaussian Process Models for Sequential Design and Optimization , 2009, 0909.5262.

[18]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[19]  Radford M. Neal.  Regression and Classification Using Gaussian Process Priors , 2009 .

[20]  Charles Audet,et al.  Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems , 2008 .

[21]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[22]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[23]  Alan E. Gelfand,et al.  Multivariate Spatial Modeling for Geostatistical Data Using Convolved Covariance Functions , 2007 .

[24]  P. Diggle,et al.  Model‐based geostatistics , 2007 .

[25]  Anton Schwaighofer,et al.  Learning Gaussian processes from multiple tasks , 2005, ICML.

[26]  W. Michael Conklin,et al.  Multivariate Bayesian Statistics: Models for Source Separation and Signal Unmixing , 2005, Technometrics.

[27]  Robert B. Gramacy,et al.  Bayesian treed gaussian process models , 2005 .

[28]  C. F. Sirmans,et al.  Nonstationary multivariate process modeling through spatially varying coregionalization , 2004 .

[29]  Sw. Banerjee,et al.  Hierarchical Modeling and Analysis for Spatial Data , 2003 .

[30]  Cass T. Miller,et al.  Optimal design for problems involving flow and transport phenomena in saturated subsurface systems , 2002 .

[31]  Geir Storvik,et al.  Particle filters for state-space models with the presence of unknown static parameters , 2002, IEEE Trans. Signal Process..

[32]  A. Gelfand,et al.  Prediction, interpolation and regression for spatially misaligned data , 2002 .

[33]  D. Higdon.  Space and Space-Time Modeling using Process Convolutions , 2002 .

[34]  M. Knott,et al.  Generalized latent trait models , 2000 .

[35]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[36]  A. Rukhin.  Matrix Variate Distributions , 1999, The Multivariate Normal Distribution.

[37]  Jun S. Liu,et al.  Sequential importance sampling for nonparametric Bayes models: The next generation , 1999 .

[38]  S. Cohn,et al.  Ooce Note Series on Global Modeling and Data Assimilation Construction of Correlation Functions in Two and Three Dimensions and Convolution Covariance Functions , 2022 .

[39]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[40]  H. Wackernagle,et al.  Multivariate geostatistics: an introduction with applications , 1998 .

[41]  L. Ryan,et al.  Latent Variable Models for Mixed Discrete and Continuous Outcomes , 1997 .

[42]  Arlen W. Harbaugh,et al.  User's documentation for MODFLOW-96, an update to the U.S. Geological Survey modular finite-difference ground-water flow model , 1996 .

[43]  Noel A Cressie,et al.  Statistics for Spatial Data, Revised Edition. , 1994 .

[44]  Noel A. C. Cressie,et al.  Statistics for Spatial Data: Cressie/Statistics , 1993 .

[45]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[46]  M. Goulard,et al.  Linear coregionalization model: Tools for estimation and choice of cross-variogram matrix , 1992 .