Machine learning-based surrogate modeling for data-driven optimization: a comparison of subset selection for regression techniques

Optimization of simulation-based or data-driven systems is a challenging task, which has attracted significant attention in the recent literature. A very efficient approach for optimizing systems without analytical expressions is through fitting surrogate models. Due to their increased flexibility, nonlinear interpolating functions, such as radial basis functions and Kriging, have been predominantly used as surrogates for data-driven optimization; however, these methods lead to complex nonconvex formulations. Alternatively, commonly used regression-based surrogates lead to simpler formulations, but they are less flexible and inaccurate if the form is not known a priori. In this work, we investigate the efficiency of subset selection regression techniques for developing surrogate functions that balance both accuracy and complexity. Subset selection creates sparse regression models by selecting only a subset of original features, which are linearly combined to generate a diverse set of surrogate models. Five different subset selection techniques are compared with commonly used nonlinear interpolating surrogate functions with respect to optimization solution accuracy, computation time, sampling requirements, and model sparsity. Our results indicate that subset selection-based regression functions exhibit promising performance when the dimensionality is low, while interpolation performs better for higher dimensional problems.

[1]  Nikolaos V. Sahinidis,et al.  A combined first-principles and data-driven approach to model building , 2015, Comput. Chem. Eng..

[2]  Andy J. Keane,et al.  Recent advances in surrogate-based optimization , 2009 .

[3]  M. Patriksson,et al.  A method for simulation based optimization using radial basis functions , 2010 .

[4]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[5]  Christodoulos A. Floudas,et al.  ARGONAUT: AlgoRithms for Global Optimization of coNstrAined grey-box compUTational problems , 2017, Optim. Lett..

[6]  Ihsan Sabuncuoglu,et al.  Simulation optimization: A comprehensive review on theory and applications , 2004 .

[7]  Nikolaos V. Sahinidis,et al.  The ALAMO approach to machine learning , 2016 .

[8]  Alan J. Miller.  Sélection of subsets of regression variables , 1984 .

[9]  Nikolaos V. Sahinidis,et al.  A polyhedral branch-and-cut approach to global optimization , 2005, Math. Program..

[10]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[11]  S. Keleş,et al.  Sparse partial least squares regression for simultaneous dimension reduction and variable selection , 2010, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[12]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Piet Demeester,et al.  A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design , 2010, J. Mach. Learn. Res..

[14]  Christodoulos A. Floudas,et al.  Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption , 2017, J. Glob. Optim..

[15]  Shuichi Kawano,et al.  Sparse principal component regression with adaptive loading , 2014, Comput. Stat. Data Anal..

[16]  Christodoulos A Floudas,et al.  Princeton_TIGRESS 2.0: High refinement consistency and net gains through support vector machines and molecular dynamics in double‐blind predictions during the CASP11 experiment , 2017, Proteins.

[17]  Hsien-Chie Cheng,et al.  Assessing a Response Surface-Based Optimization Approach for Soil Vapor Extraction System Design , 2009 .

[18]  Trevor Hastie,et al.  Statistical Learning with Sparsity: The Lasso and Generalizations , 2015 .

[19]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[20]  Ch. Venkateswarlu,et al.  Modeling and Optimization of a Pharmaceutical Formulation System Using Radial Basis Function Network , 2009, Int. J. Neural Syst..

[21]  B. Kowalski,et al.  Partial least-squares regression: a tutorial , 1986 .

[22]  Tapan Mukerji,et al.  Derivative-Free Optimization for Oil Field Operations , 2011, Computational Optimization and Applications in Engineering and Industry.

[23]  Atharv Bhosekar,et al.  Advances in surrogate based modeling, feasibility analysis, and optimization: A review , 2018, Comput. Chem. Eng..

[24]  Kazuomi Yamamoto,et al.  Efficient Optimization Design Method Using Kriging Model , 2005 .

[25]  Thomas B. Moeslund,et al.  Greedy vs. L1 Convex Optimization in Sparse Coding: Comparative Study in Abnormal Event Detection , 2015, ICML 2015.

[26]  Christine A. Shoemaker,et al.  Constrained Global Optimization of Expensive Black Box Functions Using Radial Basis Functions , 2005, J. Glob. Optim..

[27]  Ishan Bajaj,et al.  A trust region-based two phase algorithm for constrained black-box and grey-box optimization with infeasible initial point , 2017, Comput. Chem. Eng..

[28]  L. Durlofsky,et al.  Use of reduced-order models in well control optimization , 2017 .

[29]  Christodoulos A. Floudas,et al.  Global optimization of grey-box computational systems using surrogate functions and application to highly constrained oil-field operations , 2018, Comput. Chem. Eng..

[30]  Yunqian Ma,et al.  Practical selection of SVM parameters and noise estimation for SVM regression , 2004, Neural Networks.

[31]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[32]  Nikolaos V. Sahinidis,et al.  Simulation optimization: a review of algorithms and applications , 2014, 4OR.

[33]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[34]  A. Liwo,et al.  Protein structure prediction by global optimization of a potential energy function. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[35]  Marianthi G. Ierapetritou,et al.  Dynamic Data-Driven Modeling of Pharmaceutical Processes , 2011 .

[36]  D. Maguire.  The raster GIS design model: a profile of ERDAS , 1992 .

[37]  Matthew J. Realff,et al.  Metamodeling Approach to Optimization of Steady-State Flowsheet Simulations: Model Generation , 2002 .

[38]  Wei Gong,et al.  An evaluation of adaptive surrogate modeling based optimization with two benchmark problems , 2014, Environ. Model. Softw..

[39]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[40]  Randy J. Read,et al.  Improved molecular replacement by density- and energy-guided protein structure optimization , 2011, Nature.

[41]  Victor M. Zavala,et al.  Optimization formulations for multi-product supply chain networks , 2017, Comput. Chem. Eng..

[42]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[43]  David C. Miller,et al.  Learning surrogate models for simulation‐based optimization , 2014 .

[44]  Frank Pettersson,et al.  Optimization of a small-scale LNG supply chain , 2018 .

[45]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[46]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[47]  Jianhua Guo,et al.  Feature subset selection using naive Bayes for text classification , 2015, Pattern Recognit. Lett..

[48]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[49]  L. Durlofsky,et al.  A derivative-free methodology with local and global search for the constrained joint optimization of well locations and controls , 2014, Computational Geosciences.