论文信息 - On the use of metaheuristics in hyperparameters optimization of gaussian processes

On the use of metaheuristics in hyperparameters optimization of gaussian processes

Due to difficulties such as multiple local optima and flat landscape, it is suggested to use global optimization techniques to discover the global optimum of the auxiliary optimization problem of finding good Gaussian Processes (GP) hyperparameters. We investigated the performance of genetic algorithms (GA), particle swarm optimization (PSO), differential evolution (DE), and covariance matrix adaptation evolution strategy (CMA-ES) for optimizing hyperparameters of GP. The study was performed on two artificial problems and also one real-world problem. From the results, we observe that PSO, CMA-ES, and DE/local-to-best/1 consistently outperformed two variants of GA and DE/rand/1 with per-generation-dither on all problems. In particular, CMA-ES is an attractive method since it is quasi-parameter free and it also demonstrates good exploitative and explorative power on optimizing the hyperparameters.

[1] Hao Wang,et al. Optimally Weighted Cluster Kriging for Big Data Regression , 2015, IDA.

[2] Nando de Freitas,et al. Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[3] Pramudita Satria Palar,et al. A comparative study of local search within a surrogate-assisted multi-objective memetic algorithm framework for expensive problems , 2016, Appl. Soft Comput..

[4] Carl E. Rasmussen,et al. Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[5] G. Gary Wang,et al. Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.

[6] Petros Koumoutsakos,et al. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[7] Bernhard Sendhoff,et al. Evolution by Adapting Surrogates , 2013, Evolutionary Computation.

[8] Richard P. Dwight,et al. Exploiting Adjoint Derivatives in High-Dimensional Metamodels , 2015 .

[9] Joaquim R. R. A. Martins,et al. Surrogate models and mixtures of experts in aerodynamic performance prediction for mission analysis , 2014 .

[10] Timothy W. Simpson,et al. Metamodeling in Multidisciplinary Design Optimization: How Far Have We Really Come? , 2014 .

[11] Hao Wang,et al. Time complexity reduction in efficient global optimization using cluster kriging , 2017, GECCO.