A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy

Surrogate models are often used as alternatives to considerably reduce the computational burden of the expensive computer simulations that are required for engineering designs. The development of surrogate models for complex relationships between the parameters often requires the modeling of high-dimensional functions with limited information, and it is challenging to choose an effective surrogate model over the unknown design space. To this end, the ensemble models—combined with different surrogate models—offer effective solutions. This paper presents a new ensemble model based on the least squares method, which is a regularization strategy and an augmentation strategy; we call the model the regularized least squares ensemble model (RLS-EM). Three individual surrogate models—Kriging, radial basis function, and support vector regression—are used to compose the RLS-EM. Further, the weight factors are estimated by the least squares method without using the global or local error metrics, which are used in most existing methods. To solve the collinearity in the least squares calculation process, a regularization strategy and an augmentation strategy are developed. The two strategies help explore the unknown regions and improve the accuracy on one hand; on the other hand, the collinearity can be reduced, and the overfitting phenomenon that may occur can be avoided. Six numerical functions, from two-dimensional to 12-dimensional, and a computer numerical control (CNC) milling machine bed design problem are used to verify the proposed method. The results of the numerical examples show that RLS-EM saves a considerable amount of computation time while ensuring the same level of robustness and accuracy compared with other ensemble models. The RLS-EM used for the CNC milling machine bed design problem also shows good accuracy characteristics compared with other ensemble methods.

[1]  John Doherty,et al.  Offline Data-Driven Evolutionary Optimization Using Selective Surrogate Ensembles , 2019, IEEE Transactions on Evolutionary Computation.

[2]  Masoud Rais-Rohani,et al.  Ensemble of Metamodels with Optimized Weight Factors , 2008 .

[3]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox , 2002 .

[4]  Haitao Liu,et al.  A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design , 2017, Structural and Multidisciplinary Optimization.

[5]  Salvador Pintos,et al.  An Optimization Methodology of Alkaline-Surfactant-Polymer Flooding Processes Using Field Scale Numerical Simulation and Multiple Surrogates , 2004 .

[6]  Jianrong Tan,et al.  A novel support vector regression algorithm incorporated with prior knowledge and error compensation for small datasets , 2019, Neural Computing and Applications.

[7]  Xiaojian Zhou,et al.  Metamodel selection based on stepwise regression , 2016, Structural and Multidisciplinary Optimization.

[8]  Scott A. Sarra.  The Matlab Radial Basis Function Toolbox , 2017 .

[9]  Xu Liu,et al.  A Mixed-Kernel-Based Support Vector Regression Model for Automotive Body Design Optimization Under Uncertainty , 2017 .

[10]  Zuomin Dong,et al.  Hybrid and adaptive meta-model-based global optimization , 2012 .

[11]  Charles Audet,et al.  Order-based error for managing ensembles of surrogates in mesh adaptive direct search , 2018, J. Glob. Optim..

[12]  Ingo Neumann,et al.  Optimal finite element model with response surface methodology for concrete structures based on Terrestrial Laser Scanning technology , 2018 .

[13]  Guangyao Li,et al.  Crashworthiness optimization of foam-filled tapered thin-walled structure using multiple surrogate models , 2013 .

[14]  G. Gary Wang,et al.  Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.

[15]  E. Acar.  Various approaches for constructing an ensemble of metamodels using local measures , 2010 .

[16]  Eduard Muljadi,et al.  A Short-Term and High-Resolution Distribution System Load Forecasting Approach Using Support Vector Regression With Hybrid Parameters Optimization , 2018, IEEE Transactions on Smart Grid.

[17]  Matias D. Cattaneo,et al.  Inference in Linear Regression Models with Many Covariates and Heteroscedasticity , 2015, Journal of the American Statistical Association.

[18]  Keith Worden,et al.  On switching response surface models, with applications to the structural health monitoring of bridges , 2018 .

[19]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[20]  Jerome Sacks,et al.  Designs for Computer Experiments , 1989 .

[21]  G. G. Wang,et al.  Adaptive Response Surface Method Using Inherited Latin Hypercube Design Points , 2003 .

[22]  Ying Wang,et al.  Prediction of Moment Redistribution in Statically Indeterminate Reinforced Concrete Structures Using Artificial Neural Network and Support Vector Regression , 2018, Applied Sciences.

[23]  Jack P. C. Kleijnen,et al.  Stochastic Intrinsic Kriging for Simulation Metamodelling , 2015 .

[24]  Zuomin Dong,et al.  Hybrid surrogate-based optimization using space reduction (HSOSR) for expensive black-box functions , 2018, Appl. Soft Comput..

[25]  Dong-Hoon Choi,et al.  Pointwise ensemble of meta-models using v nearest points cross-validation , 2014 .

[26]  Rajkumar Roy,et al.  Recent advances in engineering design optimisation: Challenges and future trends , 2008 .

[27]  Shengli Xu,et al.  Optimal Weighted Pointwise Ensemble of Radial Basis Functions with Different Basis Functions , 2016 .

[28]  Jie Zhang,et al.  An Advanced and Robust Ensemble Surrogate Model: Extended Adaptive Hybrid Functions , 2018 .

[29]  Yan Wang,et al.  An active learning radial basis function modeling method based on self-organization maps for simulation-based design problems , 2017, Knowl. Based Syst..

[30]  Xiao Jian Zhou,et al.  Ensemble of surrogates with recursive arithmetic average , 2011 .

[31]  Wallace G. Ferreira,et al.  Ensemble of metamodels: the augmented least squares approach , 2016 .

[32]  Jinping Ou,et al.  Optimization Design of Coupling Beam Metal Damper in Shear Wall Structures , 2017 .

[33]  Pengcheng Ye,et al.  Ensemble of surrogate based global optimization methods using hierarchical design space reduction , 2018 .

[34]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[35]  H. Fang,et al.  Global response approximation with radial basis functions , 2006 .

[36]  R. Haftka,et al.  Ensemble of surrogates , 2007 .

[37]  Qing Yang,et al.  Energy Evaluation of Triggering Soil Liquefaction Based on the Response Surface Method , 2019, Applied Sciences.

[38]  R. Brereton,et al.  Support vector machines for classification and regression. , 2010, The Analyst.

[39]  Andy J. Keane,et al.  Recent advances in surrogate-based optimization , 2009 .

[40]  Yi Lv,et al.  Vibro-Acoustic Optimization Study for the Volute Casing of a Centrifugal Fan , 2019 .

[41]  Ji Pei,et al.  Application of different surrogate models on the optimization of centrifugal pump , 2016 .

[42]  Jie Hu,et al.  Multivariable case-based reason adaptation based on multiple-output support vector regression with similarity-related weight for parametric mechanical design , 2018 .

[43]  T. Simpson,et al.  Use of Kriging Models to Approximate Deterministic Computer Models , 2005 .

[44]  Atharv Bhosekar,et al.  Advances in surrogate based modeling, feasibility analysis, and optimization: A review , 2018, Comput. Chem. Eng..

[45]  Jack P. C. Kleijnen,et al.  Regression and Kriging metamodels with their experimental designs in simulation: A review , 2017, Eur. J. Oper. Res..

[46]  Hyung-Jo Jung,et al.  The Multiple-Update-Infill Sampling Method Using Minimum Energy Design for Sequential Surrogate Modeling , 2018 .

[47]  R. Haftka,et al.  Multiple surrogates: how cross-validation errors can help us to obtain the best predictor , 2009 .

[48]  Carlos Alberto Flesch,et al.  Modeling the leadership - project performance relation: radial basis function, Gaussian and Kriging methods as alternatives to linear regression , 2013, Expert Syst. Appl..

[49]  Liang Gao,et al.  Ensemble of surrogates with hybrid method using global and local measures for engineering design , 2018 .

[50]  G. Wen,et al.  Multiobjective crashworthiness optimization design of functionally graded foam-filled tapered tube based on dynamic ensemble metamodel , 2014 .

[51]  Ping Yao,et al.  Study on the wire feed speed prediction of double-wire-pulsed MIG welding based on support vector machine regression , 2015 .