First Investigations on Noisy Model-Based Multi-objective Optimization

In many real-world applications concerning multi-objective optimization, the true objective functions are not observable. Instead, only noisy observations are available. In recent years, the interest in the effect of such noise in evolutionary multi-objective optimization EMO has increased and many specialized algorithms have been proposed. However, evolutionary algorithms are not suitable if the evaluation of the objectives is expensive and only a small budget is available. One popular solution is to use model-based multi-objective optimization MBMO techniques. In this paper, we present a first investigation on noisy MBMO. For this purpose we collect several noise handling strategies from the field of EMO and adapt them for MBMO algorithms. We compare the performance of those strategies in two benchmark situations: Firstly, we perform a purely artificial benchmark using homogeneous Gaussian noise. Secondly, we choose a setting from the field of machine learning, where the structure of the underlying noise is unknown.

[1]  Benjamin W. Wah,et al.  Scheduling of Genetic Algorithms in a Noisy Environment , 1994, Evolutionary Computation.

[2]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[3]  Bernd Bischl,et al.  mlr Tutorial , 2016, ArXiv.

[4]  Yi-zhong Ma,et al.  Multiobjective Simulation Optimization Using Stochastic Kriging , 2016 .

[5]  Jack Kleijnen,et al.  White noise' assumptions revisited: Regression metamodels & experimental design in practice , 2006, Proceedings of the 2006 Winter Simulation Conference.

[6]  Bernd Bischl,et al.  Multi-objective parameter configuration of machine learning algorithms using model-based optimization , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).

[7]  Joshua D. Knowles,et al.  Multiobjective Optimization on a Budget of 250 Evaluations , 2005, EMO.

[8]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[9]  Bernd Bischl,et al.  A comparative study on large scale kernelized support vector machines , 2016, Adv. Data Anal. Classif..

[10]  Joshua D. Knowles,et al.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems , 2006, IEEE Transactions on Evolutionary Computation.

[11]  Jonathan E. Fieldsend,et al.  The Rolling Tide Evolutionary Algorithm: A Multiobjective Optimizer for Noisy Optimization Problems , 2015, IEEE Transactions on Evolutionary Computation.

[12]  Qingfu Zhang,et al.  Multiobjective optimization Test Instances for the CEC 2009 Special Session and Competition , 2009 .

[13]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[14]  Carlos A. Coello Coello,et al.  Using the Averaged Hausdorff Distance as a Performance Measure in Evolutionary Multiobjective Optimization , 2012, IEEE Transactions on Evolutionary Computation.

[15]  Bernd Bischl,et al.  Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark , 2015, EMO.

[16]  Wolfgang Ponweiser,et al.  Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted -Metric Selection , 2008, PPSN.

[17]  Robert Ivor John,et al.  Evolutionary optimisation of noisy multi-objective problems using confidence-based dynamic resampling , 2010, Eur. J. Oper. Res..

[18]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[19]  Bernd Bischl,et al.  Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation , 2012, Evolutionary Computation.

[20]  Bernd Bischl,et al.  BatchJobs and BatchExperiments: Abstraction Mechanisms for Using R in Batch Environments , 2015 .

[21]  Thomas Bäck,et al.  Efficient multi-criteria optimization on noisy machine learning problems , 2015, Appl. Soft Comput..

[22]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[23]  Bernd Bischl,et al.  mlrMBO: A Toolbox for Model-Based Optimization of Expensive Black-Box Functions , 2016 .

[24]  M. Zuluaga,et al.  ε-PAL: an active learning approach to the multi-objective optimization problem , 2016 .

[25]  Bernd Bischl,et al.  mlr: Machine Learning in R , 2016, J. Mach. Learn. Res..

[26]  Ludwig A. Hothorn,et al.  nparcomp: An R Software Package for Nonparametric Multiple Comparisons and Simultaneous Confidence Intervals , 2015 .

[27]  Daniel Hern'andez-Lobato,et al.  Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints , 2016, Neurocomputing.

[28]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[29]  Thomas Bartz-Beielstein,et al.  Simulation and Optimization of Cyclone Dust Separators , 2013 .

[30]  Andreas Krause,et al.  e-PAL: An Active Learning Approach to the Multi-Objective Optimization Problem , 2016, J. Mach. Learn. Res..

[31]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[32]  David W. Corne,et al.  Noisy Multiobjective Optimization on a Budget of 250 Evaluations , 2009, EMO.