EXTENDING CLASSICAL SURROGATE MODELING TO HIGH DIMENSIONS THROUGH SUPERVISED DIMENSIONALITY REDUCTION: A DATA-DRIVEN APPROACH

Thanks to their versatility, ease of deployment and high-performance, surrogate models have become staple tools in the arsenal of uncertainty quantification (UQ). From local interpolants to global spectral decompositions, surrogates are characterised by their ability to efficiently emulate complex computational models based on a small set of model runs used for training. An inherent limitation of many surrogate models is their susceptibility to the curse of dimensionality, which traditionally limits their applicability to a maximum of $\mathcal{O}(10^2)$ input dimensions. We present a novel approach at high-dimensional surrogate modelling that is model-, dimensionality reduction- and surrogate model- agnostic (black box), and can enable the solution of high dimensional (i.e. up to $\mathcal{O}(10^4)$) problems. After introducing the general algorithm, we demonstrate its performance by combining Kriging and polynomial chaos expansions surrogates and kernel principal component analysis. In particular, we compare the generalisation performance that the resulting surrogates achieve to the classical sequential application of dimensionality reduction followed by surrogate modelling on several benchmark applications, comprising an analytical function and two engineering applications of increasing dimensionality and complexity.

[1]  James O. Ramsay,et al.  Functional Data Analysis , 2005 .

[2]  James O. Berger,et al.  Parallel partial Gaussian process emulation for computer models with massive output , 2016 .

[3]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[4]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[5]  Bernhard Schölkopf,et al.  Learning to Find Pre-Images , 2003, NIPS.

[6]  Qiqi Wang,et al.  Erratum: Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces , 2013, SIAM J. Sci. Comput..

[7]  Jan Vybíral,et al.  Learning Functions of Few Arbitrary Linear Parameters in High Dimensions , 2010, Found. Comput. Math..

[8]  A. Kiureghian,et al.  OPTIMAL DISCRETIZATION OF RANDOM FIELDS , 1993 .

[9]  Roger G. Ghanem,et al.  Basis adaptation in homogeneous chaos spaces , 2014, J. Comput. Phys..

[10]  Geoffrey E. Hinton,et al.  Stochastic Neighbor Embedding , 2002, NIPS.

[11]  Xun Huan,et al.  Compressive sensing adaptation for polynomial chaos expansions , 2018, J. Comput. Phys..

[12]  Khachik Sargsyan,et al.  Enhancing ℓ1-minimization estimates of polynomial chaos expansions using basis selection , 2014, J. Comput. Phys..

[13]  Stefano Marelli,et al.  Data-driven polynomial chaos expansion for machine learning regression , 2018, J. Comput. Phys..

[14]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[15]  Olivier Dubrule,et al.  Cross validation of kriging in a unique neighborhood , 1983 .

[16]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[17]  B. Iooss,et al.  A Review on Global Sensitivity Analysis Methods , 2014, 1404.2405.

[18]  Bruno Sudret,et al.  Global sensitivity analysis using low-rank tensor approximations , 2016, Reliab. Eng. Syst. Saf..

[19]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[20]  François Bachoc,et al.  Cross Validation and Maximum Likelihood estimations of hyper-parameters of Gaussian processes with model misspecification , 2013, Comput. Stat. Data Anal..

[21]  W. Gautschi Orthogonal Polynomials: Computation and Approximation , 2004 .

[22]  Bruno Sudret,et al.  Polynomial meta-models with canonical low-rank approximations: Numerical insights and comparison to sparse polynomial chaos expansions , 2015, J. Comput. Phys..

[23]  Iason Papaioannou,et al.  PLS-based adaptation for efficient PCE representation in high dimensions , 2019, J. Comput. Phys..

[24]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[25]  Jon C. Helton,et al.  Sensitivity analysis of the asymptotic behavior of a model for the environmental movement of radionuclides , 1985 .

[26]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[27]  Stefano Marelli,et al.  UQLab: a framework for uncertainty quantification in MATLAB , 2014 .

[28]  Bruno Sudret,et al.  Adaptive sparse polynomial chaos expansion based on least angle regression , 2011, J. Comput. Phys..

[29]  Sylvain Arlot,et al.  A survey of cross-validation procedures for model selection , 2009, 0907.4728.

[30]  Andreas Krause,et al.  High-Dimensional Gaussian Process Bandits , 2013, NIPS.

[31]  Joe Wiart,et al.  A new surrogate modeling technique combining Kriging and polynomial chaos expansions - Application to uncertainty analysis in computational dosimetry , 2015, J. Comput. Phys..

[32]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[33]  R. Tibshirani,et al.  A bias correction for the minimum error rate in cross-validation , 2009, 0908.2904.

[34]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[35]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[36]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[37]  Andrew Gordon Wilson,et al.  Deep Kernel Learning , 2015, AISTATS.

[38]  Francesco Camastra,et al.  Data dimensionality estimation methods: a survey , 2003, Pattern Recognit..

[39]  D. Ginsbourger,et al.  Additive Covariance Kernels for High-Dimensional Gaussian Process Modeling , 2011, 1111.6233.

[40]  Michel Verleysen,et al.  The Curse of Dimensionality in Data Mining and Time Series Prediction , 2005, IWANN.

[41]  Erkki Oja,et al.  One-unit Learning Rules for Independent Component Analysis , 1996, NIPS.

[42]  R. Ghanem,et al.  Stochastic Finite Elements: A Spectral Approach , 1990 .

[43]  Karl Pearson F.R.S. LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[44]  Kilian Q. Weinberger,et al.  Learning a kernel matrix for nonlinear dimensionality reduction , 2004, ICML.

[45]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[46]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[47]  D. Xiu Numerical Methods for Stochastic Computations: A Spectral Method Approach , 2010 .

[48]  Deli Zhao,et al.  Scalable Gaussian Process Regression Using Deep Neural Networks , 2015, IJCAI.

[49]  B. Sudret,et al.  An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis , 2010 .

[50]  Quan Wang,et al.  Kernel Principal Component Analysis and its Applications in Face Recognition and Active Shape Models , 2012, ArXiv.

[51]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[52]  Ilya M. Sobol,et al.  Sensitivity Estimates for Nonlinear Mathematical Models , 1993 .

[53]  Carl E. Rasmussen,et al.  Manifold Gaussian Processes for regression , 2014, 2016 International Joint Conference on Neural Networks (IJCNN).

[54]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[55]  Mathilde Chevreuil,et al.  A Least-Squares Method for Sparse Low Rank Approximation of Multivariate Functions , 2015, SIAM/ASA J. Uncertain. Quantification.

[56]  S. Marelli,et al.  The Gaussian Process modelling module in UQLab , 2017, 1709.09382.

[57]  Ivor W. Tsang,et al.  The pre-image problem in kernel methods , 2003, IEEE Transactions on Neural Networks.

[58]  Kenji Fukumizu,et al.  Hyperparameter Selection in Kernel Principal Component Analysis , 2014, J. Comput. Sci..

[59]  Theodore B. Trafalis,et al.  Kernel principal component analysis and support vector machines for stock price prediction , 2007 .

[60]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[61]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[62]  Harvey M. Wagner,et al.  Global Sensitivity Analysis , 1995, Oper. Res..

[63]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[64]  Bruno Sudret,et al.  Comparative Study of Kriging and Support Vector Regression for Structural Engineering Applications , 2018, ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part A: Civil Engineering.