Sparse Polynomial Chaos Expansions: Literature Survey and Benchmark

Sparse polynomial chaos expansions are a popular surrogate modelling method that takes advantage of the properties of polynomial chaos expansions (PCE), the sparsity-of-effects principle, and powerful sparse regression solvers to approximate computer models with many input parameters, relying on only few model evaluations.Within the last decade, a large number of algorithms for the computation of sparse PCE have been published in the applied math and engineering literature. We present an extensive review of the existing methods and develop a framework to classify the algorithms. Furthermore, we conduct a benchmark on a selection of methods to identify which methods work best in practical applications. Comparing their accuracy on several benchmark models of varying dimension and complexity, we find that the choice of sparse regression solver and sampling scheme for the computation of a sparse PCE surrogate can make a significant difference, of up to several orders of magnitude in the resulting mean-square error. Different methods seem to be superior in different regimes of model dimensionality and experimental design size.

[1]  Roger G. Ghanem,et al.  Basis adaptation in homogeneous chaos spaces , 2014, J. Comput. Phys..

[2]  Stefano Marelli,et al.  EXTENDING CLASSICAL SURROGATE MODELING TO HIGH DIMENSIONS THROUGH SUPERVISED DIMENSIONALITY REDUCTION: A DATA-DRIVEN APPROACH , 2018 .

[3]  Naima Kaabouch,et al.  Compressive sensing: Performance comparison of sparse recovery algorithms , 2018, 2017 IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC).

[4]  R. Ghanem,et al.  Stochastic Finite Elements: A Spectral Approach , 1990 .

[5]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[6]  Amir H. Gandomi,et al.  Design of experiments for uncertainty quantification based on polynomial chaos expansion metamodels , 2020 .

[7]  J. Kiefer,et al.  Optimum Designs in Regression Problems , 1959 .

[8]  Emmanuel J. Candès,et al.  A Probabilistic and RIPless Theory of Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[9]  H. Rabitz,et al.  D-MORPH regression: application to modeling with unknown parameters more than observation data , 2010 .

[10]  S. Isukapalli.  UNCERTAINTY ANALYSIS OF TRANSPORT-TRANSFORMATION MODELS , 1999 .

[11]  Luc Pronzato,et al.  Design of computer experiments: space filling and beyond , 2011, Statistics and Computing.

[12]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing Signal Reconstruction , 2008, IEEE Transactions on Information Theory.

[13]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[14]  A. Kiureghian,et al.  OPTIMAL DISCRETIZATION OF RANDOM FIELDS , 1993 .

[15]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[16]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[17]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[18]  R. D. Cook,et al.  A Comparison of Algorithms for Constructing Exact D-Optimal Designs , 1980 .

[19]  Byeng D. Youn,et al.  Adaptive-sparse polynomial chaos expansion for reliability analysis and design of complex engineering systems , 2011 .

[20]  Zhenzhou Lu,et al.  Sparse polynomial chaos expansions for global sensitivity analysis with partial least squares and distance correlation , 2018, Structural and Multidisciplinary Optimization.

[21]  Xiu Yang,et al.  Reweighted ℓ1ℓ1 minimization method for stochastic elliptic differential equations , 2013, J. Comput. Phys..

[22]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[23]  Alexander Tarakanov,et al.  Regression-based sparse polynomial chaos for uncertainty quantification of subsurface flow models , 2019, J. Comput. Phys..

[24]  B. Sudret,et al.  An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis , 2010 .

[25]  Gary Tang,et al.  Subsampled Gauss Quadrature Nodes for Estimating Polynomial Chaos Expansions , 2014, SIAM/ASA J. Uncertain. Quantification.

[26]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[27]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[28]  Stefano Marelli,et al.  Sequential Design of Experiment for Sparse Polynomial Chaos Expansions , 2017, SIAM/ASA J. Uncertain. Quantification.

[29]  D. Xiu,et al.  STOCHASTIC COLLOCATION ALGORITHMS USING 𝓁 1 -MINIMIZATION , 2012 .

[30]  Peter E. Thornton,et al.  DIMENSIONALITY REDUCTION FOR COMPLEX MODELS VIA BAYESIAN COMPRESSIVE SENSING , 2014 .

[31]  Dongbin Xiu,et al.  On a near optimal sampling strategy for least squares polynomial regression , 2016, J. Comput. Phys..

[32]  Yoshua Bengio,et al.  Model Selection for Small Sample Regression , 2002, Machine Learning.

[33]  Paul Diaz,et al.  Sparse polynomial chaos expansions via compressed sensing and D-optimal design , 2017, Computer Methods in Applied Mechanics and Engineering.

[34]  H. Meidani,et al.  A preconditioning approach for improved estimation of sparse polynomial chaos expansions , 2017, Computer Methods in Applied Mechanics and Engineering.

[35]  David Zhang,et al.  A Survey of Sparse Representation: Algorithms and Applications , 2015, IEEE Access.

[36]  Yan Shi,et al.  An expanded sparse Bayesian learning method for polynomial chaos expansion , 2019, Mechanical Systems and Signal Processing.

[37]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[38]  Joe Wiart,et al.  Surrogate modeling based on resampled polynomial chaos expansions , 2018, Reliab. Eng. Syst. Saf..

[39]  Xun Huan,et al.  Compressive sensing adaptation for polynomial chaos expansions , 2018, J. Comput. Phys..

[40]  Bruno Sudret,et al.  Global sensitivity analysis using polynomial chaos expansions , 2008, Reliab. Eng. Syst. Saf..

[41]  Alan J. Miller,et al.  A review of some exchange algorithms for constructing discrete D-optimal designs , 1992 .

[42]  Sankaran Mahadevan,et al.  Effectively Subsampled Quadratures for Least Squares Polynomial Approximations , 2016, SIAM/ASA J. Uncertain. Quantification.

[43]  Zhenzhou Lu,et al.  An efficient and robust adaptive sampling method for polynomial chaos expansion in sparse Bayesian learning framework , 2019, Computer Methods in Applied Mechanics and Engineering.

[44]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[45]  Holger Rauhut,et al.  Sparse Legendre expansions via l1-minimization , 2012, J. Approx. Theory.

[46]  Deqiang Gan,et al.  Polynomial Chaos Expansion for Parametric Problems in Engineering Systems: A Review , 2020, IEEE Systems Journal.

[47]  Ivan V. Oseledets,et al.  Rectangular maximum-volume submatrices and their applications , 2015, ArXiv.

[48]  Bruno Sudret,et al.  Global sensitivity analysis using low-rank tensor approximations , 2016, Reliab. Eng. Syst. Saf..

[49]  Mário A. T. Figueiredo.  Adaptive Sparseness for Supervised Learning , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  Michael Ludkovski,et al.  Sequential Design for Ranking Response Surfaces , 2015, SIAM/ASA J. Uncertain. Quantification.

[51]  Bhaskar D. Rao,et al.  Perspectives on Sparse Bayesian Learning , 2003, NIPS.

[52]  Dongbin Xiu,et al.  High-Order Collocation Methods for Differential Equations with Random Inputs , 2005, SIAM J. Sci. Comput..

[53]  Akil C. Narayan,et al.  A Generalized Sampling and Preconditioning Scheme for Sparse Approximation of Polynomial Chaos Expansions , 2016, SIAM J. Sci. Comput..

[54]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[55]  O. Dykstra.  The Augmentation of Experimental Data to Maximize [X′X] , 1971 .

[56]  Xiu Yang,et al.  Sliced-Inverse-Regression-Aided Rotated Compressive Sensing Method for Uncertainty Quantification , 2017, SIAM/ASA J. Uncertain. Quantification.

[57]  Hadi Meidani,et al.  A near-optimal sampling strategy for sparse recovery of polynomial chaos expansions , 2017, J. Comput. Phys..

[58]  Fang Xu,et al.  An efficient adaptive forward–backward selection method for sparse polynomial chaos expansion , 2019, Computer Methods in Applied Mechanics and Engineering.

[59]  Ivan V. Oseledets,et al.  Gradient Descent-based D-optimal Design for the Least-Squares Polynomial Approximation , 2018, ArXiv.

[60]  Bruno Sudret,et al.  Adaptive sparse polynomial chaos expansion based on least angle regression , 2011, J. Comput. Phys..

[61]  Xun Huan,et al.  Compressive Sensing with Cross-Validation and Stop-Sampling for Sparse Polynomial Chaos Expansions , 2017, SIAM/ASA J. Uncertain. Quantification.

[62]  Alireza Doostan,et al.  Compressive sampling of polynomial chaos expansions: Convergence analysis and sampling strategies , 2014, J. Comput. Phys..

[63]  Bruno Sudret,et al.  Sparse polynomial chaos expansions and adaptive stochastic finite elements using a regression approach , 2008 .

[64]  Khachik Sargsyan,et al.  Enhancing ℓ1-minimization estimates of polynomial chaos expansions using basis selection , 2014, J. Comput. Phys..

[65]  Kyle A. Gallivan,et al.  A compressed sensing approach for partial differential equations with random input data , 2012 .

[66]  Alireza Doostan,et al.  Basis adaptive sample efficient polynomial chaos (BASE-PC) , 2017, J. Comput. Phys..

[67]  Samih Zein,et al.  An Efficient Sampling Method for Regression-Based Polynomial Chaos Expansion , 2013 .

[68]  Lawrence Carin,et al.  Bayesian Compressive Sensing , 2008, IEEE Transactions on Signal Processing.

[69]  Joe Wiart,et al.  SURROGATE MODELING OF INDOOR DOWN-LINK HUMAN EXPOSURE BASED ON SPARSE POLYNOMIAL CHAOS EXPANSION , 2019, International Journal for Uncertainty Quantification.

[70]  Matthias W. Seeger,et al.  Compressed sensing and Bayesian experimental design , 2008, ICML '08.

[71]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[72]  A. Owen.  Controlling correlations in latin hypercube samples , 1994 .

[73]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[74]  M. Lemaire,et al.  Stochastic finite element: a non intrusive approach by regression , 2006 .

[75]  Aggelos K. Katsaggelos,et al.  Bayesian Compressive Sensing Using Laplace Priors , 2010, IEEE Transactions on Image Processing.

[76]  Prasanth B. Nair,et al.  Some greedy algorithms for sparse polynomial chaos expansions , 2019, J. Comput. Phys..

[77]  O. Ernst,et al.  ON THE CONVERGENCE OF GENERALIZED POLYNOMIAL CHAOS EXPANSIONS , 2011 .

[78]  Thierry A. Mara,et al.  Bayesian sparse polynomial chaos expansion for global sensitivity analysis , 2017 .

[79]  Dongbin Xiu,et al.  Nonadaptive Quasi-Optimal Points Selection for Least Squares Linear Regression , 2016, SIAM J. Sci. Comput..

[80]  Hadi Meidani,et al.  Divide and conquer: An incremental sparsity promoting compressive sampling approach for polynomial chaos expansions , 2016, 1606.06611.

[81]  Tao Zhou,et al.  A Christoffel function weighted least squares algorithm for collocation approximations , 2014, Math. Comput..

[82]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[83]  C. Pan,et al.  Rank-Revealing QR Factorizations and the Singular Value Decomposition , 1992 .

[84]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[85]  Houman Owhadi,et al.  A non-adapted sparse approximation of PDEs with stochastic inputs , 2010, J. Comput. Phys..

[86]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[87]  Jack Xin,et al.  Minimization of ℓ1-2 for Compressed Sensing , 2015, SIAM J. Sci. Comput..

[88]  Ming Gu,et al.  Efficient Algorithms for Computing a Strong Rank-Revealing QR Factorization , 1996, SIAM J. Sci. Comput..

[89]  Iason Papaioannou,et al.  PLS-based adaptation for efficient PCE representation in high dimensions , 2019, J. Comput. Phys..

[90]  Michael D. Shields,et al.  The generalization of Latin hypercube sampling , 2015, Reliab. Eng. Syst. Saf..

[91]  Stefano Marelli,et al.  UQLab: a framework for uncertainty quantification in MATLAB , 2014 .

[92]  W. V. Harper,et al.  Sensitivity/uncertainty analysis of a borehole scenario comparing Latin Hypercube Sampling and deterministic sensitivity approaches , 1983 .

[93]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[94]  A. Doostan,et al.  Least squares polynomial chaos expansion: A review of sampling strategies , 2017, 1706.07564.

[95]  Francesco Contino,et al.  A robust and efficient stepwise regression method for building sparse polynomial chaos expansions , 2017, J. Comput. Phys..

[96]  Michael B. Wakin,et al.  An Introduction To Compressive Sampling [A sensing/sampling paradigm that goes against the common knowledge in data acquisition] , 2008 .

[97]  Zhenzhou Lu,et al.  Sparse polynomial chaos expansion based on D-MORPH regression , 2018, Appl. Math. Comput..

[98]  Robert D. Nowak,et al.  Wavelet-based image estimation: an empirical Bayes approach using Jeffrey's noninformative prior , 2001, IEEE Trans. Image Process..

[99]  Alireza Doostan,et al.  A weighted l1-minimization approach for sparse polynomial chaos expansions , 2013, J. Comput. Phys..

[100]  Michael E. Tipping,et al.  Analysis of Sparse Bayesian Learning , 2001, NIPS.

[101]  Tao Zhou,et al.  Multivariate Discrete Least-Squares Approximations with a New Type of Collocation Grid , 2014, SIAM J. Sci. Comput..

[102]  Michael E. Tipping,et al.  Fast Marginal Likelihood Maximisation for Sparse Bayesian Models , 2003 .

[103]  Sungyoung Lee,et al.  Compressive sensing: From theory to applications, a survey , 2013, Journal of Communications and Networks.

[104]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[105]  Yuhang Chen,et al.  Stochastic collocation methods via $L_1$ minimization using randomized quadratures , 2016, 1602.00995.

[106]  J. Halton.  On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals , 1960 .

[107]  Zhenzhou Lu,et al.  Adaptive sparse polynomial chaos expansions for global sensitivity analysis based on support vector regression , 2018 .

[108]  Knut Baumann,et al.  Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation , 2014, Journal of Cheminformatics.

[109]  Wei Zhao,et al.  Global sensitivity analysis with a hierarchical sparse metamodeling method , 2019, Mechanical Systems and Signal Processing.

[110]  Danny Lathouwers,et al.  Grid and basis adaptive polynomial chaos techniques for sensitivity and uncertainty analysis , 2014, J. Comput. Phys..

[111]  Alireza Doostan,et al.  Coherence motivated sampling and convergence analysis of least squares polynomial Chaos regression , 2014, 1410.1931.

[112]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .