Surrogate-Assisted Multicriteria Optimization: Complexities, Prospective Solutions, and Business Case

Complexity in solving real-world multicriteria optimization problems often stems from the fact that complex, expensive, and/or time-consuming simulation tools or physical experiments are used to evaluate solutions to a problem. In such settings, it is common to use efficient computational models, often known as surrogates or metamodels, to approximate the outcome (objective or constraint function value) of a simulation or physical experiment. The presence of multiple objective functions poses an additional layer of complexity for surrogate-assisted optimization. For example, complexities may relate to the appropriate selection of metamodels for the individual objective functions, extensive training time of surrogate models, or the optimal use of many-core computers to approximate efficiently multiple objectives simultaneously. Thinking out of the box, complexity can also be shifted from approximating the individual objective functions to approximating the entire Pareto front. This leads to further complexities, namely, how to validate statistically and apply the techniques developed to real-world problems. In this paper, we discuss emerging complexity-related topics in surrogate-assisted multicriteria optimization that may not be prevalent in nonsurrogate-assisted single-objective optimization. These complexities are motivated using several real-world problems in which the authors were involved. We then discuss several promising future research directions and prospective solutions to tackle emerging complexities in surrogate-assisted multicriteria optimization. Finally, we provide insights from an industrial point of view into how surrogate-assisted multicriteria optimization techniques can be developed and applied within a collaborative business environment to tackle real-world problems.

[1]  Wolfgang Ponweiser,et al.  On Expected-Improvement Criteria for Model-based Multi-objective Optimization , 2010, PPSN.

[2]  Donald R. Jones,et al.  Global versus local search in constrained optimization of computer models , 1998 .

[3]  Qingfu Zhang,et al.  Distributed evolutionary algorithms and their models: A survey of the state-of-the-art , 2015, Appl. Soft Comput..

[4]  Yaochu Jin,et al.  Surrogate-assisted evolutionary computation: Recent advances and future challenges , 2011, Swarm Evol. Comput..

[5]  Bernhard Sendhoff,et al.  Multi co-objective evolutionary optimization: Cross surrogate augmentation for computationally expensive problems , 2012, 2012 IEEE Congress on Evolutionary Computation.

[6]  Thomas Bäck,et al.  Mixed-integer evolution strategies for parameter optimization and their applications to medical image analysis , 2005 .

[7]  Roman Neruda,et al.  An Evolutionary Strategy for Surrogate-Based Multiobjective Optimization , 2012, 2012 IEEE Congress on Evolutionary Computation.

[8]  Qingfu Zhang,et al.  Multiobjective Optimization Problems With Complicated Pareto Sets, MOEA/D and NSGA-II , 2009, IEEE Transactions on Evolutionary Computation.

[9]  Bernhard Sendhoff,et al.  On Test Functions for Evolutionary Multi-objective Optimization , 2004, PPSN.

[10]  Milan Zeleny,et al.  Optimal system design with multiple criteria: De Novo programming approach , 1986 .

[11]  Aimin Zhou,et al.  A classification and Pareto domination based multiobjective evolutionary algorithm , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[12]  Thomas Bartz-Beielstein,et al.  A Case Study on Multi-Criteria Optimization of an Event Detection Software under Limited Budgets , 2013, EMO.

[13]  Bernd Bischl,et al.  Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark , 2015, EMO.

[14]  H. Ishibuchi,et al.  MOGA: multi-objective genetic algorithms , 1995, Proceedings of 1995 IEEE International Conference on Evolutionary Computation.

[15]  Thomas Bäck,et al.  An Archive Maintenance Scheme for Finding Robust Solutions , 2010, PPSN.

[16]  Michael T. M. Emmerich,et al.  Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels , 2006, IEEE Transactions on Evolutionary Computation.

[17]  Ashok D. Belegundu,et al.  Multi-objective optimization of laminated ceramic composites using genetic algorithms , 1994 .

[18]  Alan D. Christiansen,et al.  An empirical study of evolutionary techniques for multiobjective optimization in engineering design , 1996 .

[19]  Petros Koumoutsakos,et al.  Accelerating evolutionary algorithms with Gaussian process fitness function models , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[20]  Joshua D. Knowles,et al.  Efficient discovery of anti-inflammatory small molecule combinations using evolutionary computing , 2011, Nature chemical biology.

[21]  Carlos A. Coello Coello,et al.  THEORETICAL AND NUMERICAL CONSTRAINT-HANDLING TECHNIQUES USED WITH EVOLUTIONARY ALGORITHMS: A SURVEY OF THE STATE OF THE ART , 2002 .

[22]  António Gaspar-Cunha,et al.  A Multi-Objective Evolutionary Algorithm Using Neural Networks to Approximate Fitness Evaluations , 2005, Int. J. Comput. Syst. Signals.

[23]  Joshua D. Knowles,et al.  Multiobjective Optimization: When Objectives Exhibit Non-Uniform Latencies , 2015 .

[24]  Bernhard Sendhoff,et al.  Evolution by Adapting Surrogates , 2013, Evolutionary Computation.

[25]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[26]  Jack P. C. Kleijnen,et al.  Multivariate versus Univariate Kriging Metamodels for Multi-Response Simulation Models , 2014, Eur. J. Oper. Res..

[27]  Kalyanmoy Deb,et al.  An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints , 2014, IEEE Transactions on Evolutionary Computation.

[28]  Qingfu Zhang,et al.  Multiobjective optimization Test Instances for the CEC 2009 Special Session and Competition , 2009 .

[29]  Hao Wang,et al.  A Multicriteria Generalization of Bayesian Global Optimization , 2016, Advances in Stochastic and Deterministic Global Optimization.

[30]  Peter J. Fleming,et al.  Many-Objective Optimization: An Engineering Design Perspective , 2005, EMO.

[31]  Kamy Sepehrnoori,et al.  Tie-Simplex-Based Phase-Behavior Modeling in an IMPEC Reservoir Simulator , 2014 .

[32]  Kyriakos C. Giannakoglou,et al.  A multilevel approach to single- and multiobjective aerodynamic optimization , 2008 .

[33]  Tapabrata Ray,et al.  An Evolutionary Algorithm with Spatially Distributed Surrogates for Multiobjective Optimization , 2007, ACAL.

[34]  Domenico Quagliarella,et al.  Multipoint transonic airfoil design by means of a multiobjective genetic algorithm , 1997 .

[35]  Kyriakos C. Giannakoglou,et al.  A multi-objective metamodel-assisted memetic algorithm with strength-based local refinement , 2009 .

[36]  Stefan Wilhelm,et al.  Estimating Spatial Probit Models in R , 2013, R J..

[37]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[38]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[39]  José M. Matías Multi-output Nonparametric Regression , 2005, EPIA.

[40]  Martin Holena,et al.  Surrogate Model for Continuous and Discrete Genetic Optimization Based on RBF Networks , 2010, IDEAL.

[41]  A. Osyczka,et al.  A new method to solve generalized multicriteria optimization problems using the simple genetic algorithm , 1995 .

[42]  Frank Kursawe,et al.  A Variant of Evolution Strategies for Vector Optimization , 1990, PPSN.

[43]  Ann Nowé,et al.  Designing multi-objective multi-armed bandits algorithms: A study , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[44]  Tony R. Martinez,et al.  Heterogeneous radial basis function networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[45]  Christian Fonteix,et al.  Multicriteria optimization using a genetic algorithm for determining a Pareto set , 1996, Int. J. Syst. Sci..

[46]  Bernd Bischl,et al.  Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation , 2012, Evolutionary Computation.

[47]  Wolfgang Ponweiser,et al.  Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted -Metric Selection , 2008, PPSN.

[48]  Andreas Zell,et al.  Evolution strategies assisted by Gaussian processes with improved preselection criterion , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[49]  Kyriakos C. Giannakoglou,et al.  Design of optimal aerodynamic shapes using stochastic optimization methods and computational intelligence , 2002 .

[50]  T. T. Binh MOBES : A multiobjective evolution strategy for constrained optimization problems , 1997 .

[51]  Bernhard Sendhoff,et al.  Evolutionary Optimization with Dynamic Fidelity Computational Models , 2008, ICIC.

[52]  Liang Shi,et al.  Multiobjective GA optimization using reduced models , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[53]  Hirotaka Nakayama,et al.  Evolutionary multi-objective optimization using expected improvement and generalized DEA , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[54]  Vincenzo Catania,et al.  A Study on Evolutionary Multi-Objective Optimization with Fuzzy Approximation for Computational Expensive Problems , 2012, PPSN.

[55]  Tomoyuki Hiroyasu,et al.  DIVIDED RANGE GENETIC ALGORITHMS IN MULTIOBJECTIVE OPTIMIZATION PROBLEMS , 1999 .

[56]  Marco Laumanns,et al.  A Spatial Predator-Prey Approach to Multi-objective Optimization: A Preliminary Study , 1998, PPSN.

[57]  C. Poloni,et al.  Hybridization of a multi-objective genetic algorithm, a neural network and a classical optimizer for a complex design problem in fluid dynamics , 2000 .

[58]  Michèle Sebag,et al.  Comparison-Based Optimizers Need Comparison-Based Surrogates , 2010, PPSN.

[59]  Hugo Jair Escalante,et al.  A hybrid surrogate-based approach for evolutionary multi-objective optimization , 2013, 2013 IEEE Congress on Evolutionary Computation.

[60]  Hisashi Tamaki,et al.  Generation of a Set of Pareto-Optimal Solutions by Genetic Algorithms , 1995 .

[61]  Wei Shyy,et al.  Hydraulic Turbine Diffuser Shape Optimization by Multiple Surrogate Model Approximations of Pareto Fronts , 2007 .

[62]  Hao Wang,et al.  Optimally Weighted Cluster Kriging for Big Data Regression , 2015, IDA.

[63]  John R. Woodward,et al.  Metaheuristic Design Pattern: Surrogate Fitness Functions , 2015, GECCO.

[64]  Christina J. Hopfe,et al.  Robust multi-criteria design optimization in building design , 2012 .

[65]  John Doherty,et al.  Recurrent neural network ensembles for convergence prediction in surrogate-assisted evolutionary optimization , 2013, 2013 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE).

[66]  Kaisa Miettinen,et al.  A survey on handling computationally expensive multiobjective optimization problems using surrogates: non-nature inspired methods , 2015, Structural and Multidisciplinary Optimization.

[67]  Stefan Roth,et al.  Covariance Matrix Adaptation for Multi-objective Optimization , 2007, Evolutionary Computation.

[68]  Qingfu Zhang,et al.  Expensive Multiobjective Optimization by MOEA/D With Gaussian Process Model , 2010, IEEE Transactions on Evolutionary Computation.

[69]  Antanas Žilinskas,et al.  On the Statistical Models-Based Multi-objective Optimization , 2014 .

[70]  Roman Neruda,et al.  Aggregate meta-models for evolutionary multiobjective and many-objective optimization , 2013, Neurocomputing.

[71]  Gary B. Lamont,et al.  Multiobjective evolutionary algorithm test suites , 1999, SAC '99.

[72]  Ingo Rechenberg,et al.  Case studies in evolutionary experimentation and computation , 2000 .

[73]  Shengxiang Yang,et al.  Evolutionary dynamic optimization: A survey of the state of the art , 2012, Swarm Evol. Comput..

[74]  A. Eiben,et al.  A multi-sexual genetic algorithm for multiobjective optimization , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[75]  Bernhard Sendhoff,et al.  Generalizing Surrogate-Assisted Evolutionary Computation , 2010, IEEE Transactions on Evolutionary Computation.

[76]  Xu Xu,et al.  Surrogate Models for Mixed Discrete-Continuous Variables , 2014, Constraint Programming and Decision Making.

[77]  Gary B. Lamont,et al.  Multiobjective evolutionary algorithms: classifications, analyses, and new innovations , 1999 .

[78]  Robert B. Gramacy,et al.  Bayesian treed gaussian process models , 2005 .

[79]  Roman Neruda,et al.  Hypervolume-based local search in multi-objective evolutionary optimization , 2014, GECCO.

[80]  Chee Keong Kwoh,et al.  Feasibility Structure Modeling: An Effective Chaperone for Constrained Memetic Algorithms , 2010, IEEE Transactions on Evolutionary Computation.

[81]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[82]  Tapabrata Ray,et al.  Surrogate assisted Simulated Annealing (SASA) for constrained multi-objective optimization , 2010, IEEE Congress on Evolutionary Computation.

[83]  Markus Olhofer,et al.  Reference vector based a posteriori preference articulation for evolutionary multiobjective optimization , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[84]  Manuel Valenzuela-Rendón,et al.  A Non-Generational Genetic Algorithm for Multiobjective Optimization , 1997, ICGA.

[85]  Qingfu Zhang,et al.  MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition , 2007, IEEE Transactions on Evolutionary Computation.

[86]  Peter J. Fleming,et al.  An Overview of Evolutionary Algorithms in Multiobjective Optimization , 1995, Evolutionary Computation.

[87]  Lamjed Ben Said,et al.  Steady state IBEA assisted by MLP neural networks for expensive multi-objective optimization problems , 2014, GECCO.

[88]  Masahiro Tanaka,et al.  GA-based decision support system for multicriteria optimization , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[89]  Thomas Bäck,et al.  Efficient multi-criteria optimization on noisy machine learning problems , 2015, Appl. Soft Comput..

[90]  Ivor W. Tsang,et al.  Pareto Rank Learning in Multi-objective Evolutionary Algorithms , 2012, 2012 IEEE Congress on Evolutionary Computation.

[91]  Andrzej Osyczka,et al.  7 – Multicriteria optimization for engineering design , 1985 .

[92]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[93]  Andreas Krause,et al.  Active Learning for Multi-Objective Optimization , 2013, ICML.

[94]  Enrique Alba,et al.  Parallel metaheuristics: recent advances and new trends , 2012, Int. Trans. Oper. Res..

[95]  Carlos Kavka,et al.  Towards a Standard Approach for Optimization in Science and Engineering , 2013, ICSOFT.

[96]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[97]  Kalyanmoy Deb,et al.  Multi-objective Genetic Algorithms: Problem Difficulties and Construction of Test Problems , 1999, Evolutionary Computation.

[98]  Shigeru Obayashi,et al.  Comparison of the criteria for updating Kriging response surface models in multi-objective optimization , 2012, 2012 IEEE Congress on Evolutionary Computation.

[99]  Jonathan A. Wright,et al.  Constrained, mixed-integer and multi-objective optimisation of building designs by NSGA-II with fitness approximation , 2015, Appl. Soft Comput..

[100]  Virginia Torczon,et al.  Using approximations to accelerate engineering design optimization , 1998 .

[101]  Kok Wai Wong,et al.  Surrogate-Assisted Evolutionary Optimization Frameworks for High-Fidelity Engineering Design Problems , 2005 .

[102]  Bernd Bischl,et al.  PROGRESS: Progressive Reinforcement-Learning-Based Surrogate Selection , 2013, LION.

[103]  M Monz,et al.  Pareto navigation—algorithmic foundation of interactive multi-criteria IMRT planning , 2008, Physics in medicine and biology.

[104]  Tomi Haanpää,et al.  Approximation method for computationally expensive nonconvex multiobjective optimization problems , 2012 .

[105]  Tom Dhaene,et al.  Fast calculation of multiobjective probability of improvement and expected improvement criteria for Pareto optimization , 2014, J. Glob. Optim..

[106]  Carlos A. Coello Coello,et al.  Multi-objective airfoil shape optimization using a multiple-surrogate approach , 2012, 2012 IEEE Congress on Evolutionary Computation.

[107]  Joshua D. Knowles Closed-loop evolutionary multiobjective optimization , 2009, IEEE Computational Intelligence Magazine.

[108]  R. Lyndon While,et al.  A review of multiobjective test problems and a scalable test problem toolkit , 2006, IEEE Transactions on Evolutionary Computation.

[109]  Rajkumar Roy,et al.  Evolutionary-based techniques for real-life optimisation: development and testing , 2002, Appl. Soft Comput..

[110]  Roman Neruda,et al.  ASM-MOMA: Multiobjective memetic algorithm with aggregate surrogate model , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[111]  Marco Laumanns,et al.  Scalable multi-objective optimization test problems , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[112]  Kalyanmoy Deb,et al.  On the performance of classification algorithms for learning Pareto-dominance relations , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[113]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[114]  Oussama Khatib,et al.  The explicit dynamic model and inertial parameters of the PUMA 560 arm , 1986, Proceedings. 1986 IEEE International Conference on Robotics and Automation.

[115]  Nicola Beume,et al.  An EMO Algorithm Using the Hypervolume Measure as Selection Criterion , 2005, EMO.

[116]  R. Lyndon While,et al.  A Scalable Multi-objective Test Problem Toolkit , 2005, EMO.

[117]  Michael T. M. Emmerich,et al.  Faster Exact Algorithms for Computing Expected Hypervolume Improvement , 2015, EMO.

[118]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[119]  Peter J. Fleming,et al.  Multiobjective genetic algorithms made easy: selection sharing and mating restriction , 1995 .

[120]  S. Rajeev,et al.  Discrete Optimization of Structures Using Genetic Algorithms , 1992 .

[121]  Hirotaka Nakayama,et al.  A generalized model for data envelopment analysis , 2004, Eur. J. Oper. Res..

[122]  Alessandro Turco,et al.  Metamodels for Fast Multi-objective Optimization: Trading Off Global Exploration and Local Exploitation , 2010, SEAL.

[123]  T. Santner,et al.  Computer experiments: multiobjective optimization and sensitivity analysis , 2011 .

[124]  Kalyanmoy Deb,et al.  Constrained Test Problems for Multi-objective Evolutionary Optimization , 2001, EMO.

[125]  Ashok Dhondu Belegundu,et al.  A Study of Mathematical Programming Methods for Structural Optimization , 1985 .

[126]  Andy J. Keane,et al.  Multi-Objective Optimization Using Surrogates , 2010 .

[127]  Shigeru Obayashi,et al.  Efficient global optimization (EGO) for multi-objective problem and data mining , 2005, 2005 IEEE Congress on Evolutionary Computation.

[128]  J. Arora,et al.  A study of mathematical programmingmethods for structural optimization. Part II: Numerical results , 1985 .

[129]  Michael T. M. Emmerich,et al.  Hypervolume-based expected improvement: Monotonicity properties and exact computation , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[130]  Aimin Zhou,et al.  A Multioperator Search Strategy Based on Cheap Surrogate Models for Evolutionary Optimization , 2015, IEEE Transactions on Evolutionary Computation.

[131]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[132]  Joshua D. Knowles,et al.  'Hang On a Minute': Investigations on the Effects of Delayed Objective Functions in Multiobjective Optimization , 2013, EMO.

[133]  Saúl Zapotecas Martínez,et al.  Combining surrogate models and local search for dealing with expensive multi-objective optimization problems , 2013, 2013 IEEE Congress on Evolutionary Computation.

[134]  Kalyanmoy Deb,et al.  Multi-objective test problems, linkages, and evolutionary methodologies , 2006, GECCO.

[135]  Thomas Bäck,et al.  Metamodel-assisted mixed integer evolution strategies and their application to intravascular ultrasound image analysis , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[136]  Suzanne S. Farid,et al.  Tuning Evolutionary Multiobjective Optimization for Closed-Loop Estimation of Chromatographic Operating Conditions , 2014, PPSN.

[137]  Kathrin Klamroth,et al.  Pareto navigator for interactive nonlinear multiobjective optimization , 2010, OR Spectr..

[138]  Ahmed Kattan,et al.  Geometric Generalisation of Surrogate Model Based Optimisation to Combinatorial Spaces , 2011, EvoCOP.

[139]  Thomas Bartz-Beielstein,et al.  Distance Measures for Permutations in Combinatorial Efficient Global Optimization , 2014, PPSN.

[140]  Jeng-Shyang Pan,et al.  A new fitness estimation strategy for particle swarm optimization , 2013, Inf. Sci..

[141]  David A. Romero A multi-stage, multi-response Bayesian methodology for surrogate modeling in engineering design , 2008 .

[142]  Filip De Turck,et al.  Evolutionary Model Type Selection for Global Surrogate Modeling , 2009, J. Mach. Learn. Res..

[143]  Bernhard Sendhoff,et al.  A Multiobjective Evolutionary Algorithm Using Gaussian Process-Based Inverse Modeling , 2015, IEEE Transactions on Evolutionary Computation.

[144]  Roberto Battiti,et al.  Active Learning of Pareto Fronts , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[145]  Kaisa Miettinen,et al.  PAINT: Pareto front interpolation for nonlinear multiobjective optimization , 2012, Comput. Optim. Appl..

[146]  R. DeVor,et al.  An application of multiple criteria decision making principles for planning machining operations , 1984 .

[147]  Michèle Sebag,et al.  A mono surrogate for multiobjective optimization , 2010, GECCO '10.

[148]  Thomas Bäck,et al.  Mixed Integer Evolution Strategies for Parameter Optimization , 2013, Evolutionary Computation.

[149]  Michèle Sebag,et al.  Dominance-Based Pareto-Surrogate for Multi-Objective Optimization , 2010, SEAL.

[150]  Joshua D. Knowles,et al.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems , 2006, IEEE Transactions on Evolutionary Computation.

[151]  Juhani Koski,et al.  Multicriteria Optimization — Fundamentals and Motivation , 1990 .

[152]  Jeremy E. Oakley,et al.  Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures , 2013, Technometrics.

[153]  Hirotaka Nakayama,et al.  Meta-Modeling in Multiobjective Optimization , 2008, Multiobjective Optimization.

[154]  Hisao Ishibuchi,et al.  Behavior of Evolutionary Many-Objective Optimization , 2008, Tenth International Conference on Computer Modeling and Simulation (uksim 2008).

[155]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..

[156]  Hajime Kita,et al.  Multi-objective optimization by genetic algorithms: a review , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[157]  Thomas Bartz-Beielstein,et al.  Efficient global optimization for combinatorial problems , 2014, GECCO.

[158]  Richard Flavell Approximate Matrix Inversion , 1977 .

[159]  Hisao Ishibuchi,et al.  Evolutionary many-objective optimization: A short review , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[160]  David Ginsbourger,et al.  Quantifying uncertainty on Pareto fronts with Gaussian process conditional simulations , 2015, Eur. J. Oper. Res..

[161]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[162]  Carlos A. Coello Coello,et al.  A Review of Techniques for Handling Expensive Functions in Evolutionary Multi-Objective Optimization , 2010 .

[163]  A. Vicini,et al.  Sub-population policies for a parallel multiobjective genetic algorithm with applications to wing design , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).