A Nested Weighted Tchebycheff Multi-Objective Bayesian Optimization Approach for Flexibility of Unknown Utopia Estimation in Expensive Black-box Design Problems

We propose a nested weighted Tchebycheff Multi-objective Bayesian optimization framework where we build a regression model selection procedure from an ensemble of models, towards better estimation of the uncertain parameters of the weighted-Tchebycheff expensive black-box multi-objective function. In existing work, a weighted Tchebycheff MOBO approach has been demonstrated which attempts to estimate the unknown utopia in formulating acquisition function, through calibration using a priori selected regression model. However, the existing MOBO model lacks flexibility in selecting the appropriate regression models given the guided sampled data and therefore, can under-fit or over-fit as the iterations of the MOBO progress, reducing the overall MOBO performance. As it is too complex to a priori guarantee a best model in general, this motivates us to consider a portfolio of different families of predictive models fitted with current training data, guided by the WTB MOBO; the best model is selected following a user-defined prediction root mean-square-error-based approach. The proposed approach is implemented in optimizing a multi-modal benchmark problem and a thin tube design under constant loading of temperature-pressure, with minimizing the risk of creep-fatigue failure and design cost. Finally, the nested weighted Tchebycheff MOBO model performance is compared with different MOBO frameworks with respect to accuracy in parameter estimation, Pareto-optimal solutions and function evaluation cost. This method is generalized enough to consider different families of predictive models in the portfolio for best model selection, where the overall design architecture allows for solving any high-dimensional (multiple functions) complex black-box problems and can be extended to any other global criterion multi-objective optimization methods where prior knowledge of utopia is required.

[1]  Ping Zhu,et al.  Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design , 2019, Journal of Mechanical Design.

[2]  C. Holmes,et al.  Bayesian auxiliary variable models for binary and multinomial regression , 2006 .

[3]  Harold J. Kushner,et al.  A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise , 1964 .

[4]  Wenqiang Zhang,et al.  Improved Vector Evaluated Genetic Algorithm with Archive for Solving Multiobjective PPS Problem , 2010, 2010 International Conference on E-Product E-Service and E-Entertainment.

[5]  Lai-Wan Chan,et al.  Support Vector Machine Regression for Volatile Stock Market Prediction , 2002, IDEAL.

[6]  D. Lizotte.  Practical bayesian optimization , 2008 .

[7]  Irem Y. Tumer,et al.  Model Validation in Early Phase of Designing Complex Engineered Systems , 2018, Volume 2A: 44th Design Automation Conference.

[8]  Thurston Sexton,et al.  Learning an Optimization Algorithm through Human Design Iterations , 2016, 1608.06984.

[9]  D. Dennis,et al.  A statistical method for global optimization , 1992, [Proceedings] 1992 IEEE International Conference on Systems, Man, and Cybernetics.

[10]  Jasbir S. Arora,et al.  Survey of multi-objective optimization methods for engineering , 2004 .

[11]  Gary William Flake,et al.  Efficient SVM Regression Training with SMO , 2002, Machine Learning.

[12]  Andrey Pepelyshev,et al.  The Role of the Nugget Term in the Gaussian Process Method , 2010, 1005.4385.

[13]  Taejung Yeo,et al.  A novel multistage Support Vector Machine based approach for Li ion battery remaining useful life estimation , 2015 .

[14]  Xiaobo Zhou,et al.  Global Sensitivity Analysis , 2017, Encyclopedia of GIS.

[15]  L. Thurstone.  A law of comparative judgment. , 1994 .

[16]  Wei Chen,et al.  A better understanding of model updating strategies in validating engineering models , 2009 .

[17]  Kwei-Jay Lin,et al.  A theory of lexicographic multi-criteria optimization , 1996, Proceedings of ICECCS '96: 2nd IEEE International Conference on Engineering of Complex Computer Systems (held jointly with 6th CSESAW and 4th IEEE RTAW).

[18]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[19]  Paris Perdikaris,et al.  Multifidelity and Multiscale Bayesian Framework for High-Dimensional Engineering Design and Calibration , 2019, Journal of Mechanical Design.

[20]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[21]  Edward Lloyd Snelson,et al.  Flexible and efficient Gaussian process models for machine learning , 2007 .

[22]  Marcus R. Frean,et al.  Using Gaussian Processes to Optimize Expensive Functions , 2008, Australasian Conference on Artificial Intelligence.

[23]  Søren Nymand Lophaven,et al.  DACE - A Matlab Kriging Toolbox , 2002 .

[24]  T. Ouarda,et al.  Bayesian multivariate linear regression with application to change point models in hydrometeorological variables , 2007 .

[25]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[26]  Christopher Hoyle,et al.  An Approach to Bayesian Optimization for Design Feasibility Check on Discontinuous Black-Box Functions , 2020, Journal of Mechanical Design.

[27]  Zi Wang,et al.  Max-value Entropy Search for Efficient Bayesian Optimization , 2017, ICML.

[28]  Wei Chu,et al.  Extensions of Gaussian Processes for Ranking : Semi-supervised and Active Learning , 2005 .

[29]  Jiří Holík,et al.  Problems of a Utopia Point Setting in Transformation of Individual Objective Functions in Multi-Objective Optimization , 2019 .

[30]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[31]  Yan Wang,et al.  A New Multi-Objective Bayesian Optimization Formulation With the Acquisition Function for Convergence and Diversity , 2020 .

[32]  Improved rank‐niche evolution strategy algorithm for constrained multiobjective optimization , 2008 .

[33]  S. Suresh,et al.  Revisiting norm optimization for multi-objective black-box problems: a finite-time analysis , 2018, J. Glob. Optim..

[34]  C.A. Coello Coello,et al.  MOPSO: a proposal for multiple objective particle swarm optimization , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[35]  Wasim Akram Mandal.  Weighted Tchebycheff Optimization Technique Under Uncertainty , 2020 .

[36]  Nando de Freitas,et al.  A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot , 2009, Auton. Robots.

[37]  Bruce E. Ankenman,et al.  Comparison of Gaussian process modeling software , 2016, 2016 Winter Simulation Conference (WSC).

[38]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[39]  Yongming Han,et al.  Review: Multi-objective optimization methods and application in energy saving , 2017 .

[40]  Svetha Venkatesh,et al.  Multi-objective Bayesian optimisation with preferences over objectives , 2019, NeurIPS.

[41]  Peter Challenor,et al.  Computational Statistics and Data Analysis the Effect of the Nugget on Gaussian Process Emulators of Computer Models , 2022 .

[42]  Vlad M. Cora.  Model-Based Active Learning in Hierarchical Policies , 2008 .

[43]  Daniel Hern'andez-Lobato,et al.  Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints , 2016, Neurocomputing.

[44]  Michael T. M. Emmerich,et al.  Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels , 2006, IEEE Transactions on Evolutionary Computation.

[45]  Ajay K. Ray,et al.  APPLICATIONS OF MULTIOBJECTIVE OPTIMIZATION IN CHEMICAL ENGINEERING , 2000 .

[46]  Elaine Martin,et al.  Bayesian linear regression and variable selection for spectroscopic calibration. , 2009, Analytica chimica acta.

[47]  Tao Wang,et al.  Automatic Gait Optimization with Gaussian Process Regression , 2007, IJCAI.

[48]  Thomas Bäck,et al.  Multi-Objective Bayesian Global Optimization using expected hypervolume improvement gradient , 2019, Swarm Evol. Comput..

[49]  F. Mosteller.  Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations , 1951 .

[50]  J. Bree.  Elastic-plastic behaviour of thin tubes subjected to internal pressure and intermittent high-heat fluxes with application to fast-nuclear-reactor fuel elements , 1967 .

[51]  B. Paul,et al.  Feasibility of Using Diffusion Bonding for Producing Hybrid Printed Circuit Heat Exchangers for Nuclear Energy Applications , 2018 .

[52]  Joshua D. Knowles,et al.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems , 2006, IEEE Transactions on Evolutionary Computation.

[53]  Wei Xing,et al.  Shared-Gaussian Process: Learning Interpretable Shared Hidden Structure Across Data Spaces for Design Space Analysis and Exploration , 2020 .

[54]  V. Bowman.  On the Relationship of the Tchebycheff Norm and the Efficient Frontier of Multiple-Criteria Objectives , 1976 .

[55]  David L. Olson.  Tchebycheff norms in multi-objective linear programming , 1993 .

[56]  Svetha Venkatesh,et al.  Expected Hypervolume Improvement with Constraints , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).