Batch Bayesian optimization via adaptive local search

Bayesianoptimization (BO) provides an efficient tool for solving the black-box global optimization problems. Under situations where multiple points can be evaluated simultaneously, batch Bayesian optimization has been a popular extension by taking full use of the computational and experimental resources. In this paper, an adaptive local search strategy is investigated to select batch points for Bayesian optimization. First, multi-start strategy and gradient-based optimization method are combined to maximize the acquisition function. Then, an automatic cluster approach (e.g., X-means) is applied to adaptively identify the acquisition function’s local maxima from the gradient-based optimization results. Third, the Bayesian stopping criterion is utilized to guarantee all the local maxima can be obtained theoretically. Moreover, the lower bound confidence criterion and frontend truncation operation are employed to select the most promising local maxima as batch points. Extensive evaluations on various synthetic functions and two hyperparameter tuning problems for deep learning models are utilized to verify the proposed method.

[1]  Wei-Yun Yau,et al.  Structured AutoEncoders for Subspace Clustering , 2018, IEEE Transactions on Image Processing.

[2]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[3]  O. A. Elwakeil,et al.  Global optimization methods for engineering applications: A review , 1995 .

[4]  Sungyoung Lee,et al.  Gaussian process for predicting CPU utilization and its application to energy efficiency , 2015, Applied Intelligence.

[5]  R. Lyons Determinantal probability measures , 2002, math/0204325.

[6]  Nicolas Vayatis,et al.  Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration , 2013, ECML/PKDD.

[7]  T. Shirai,et al.  Random point fields associated with certain Fredholm determinants I: fermion, Poisson and boson point processes , 2003 .

[8]  Jeong‐Soo Park Optimal Latin-hypercube designs for computer experiments , 1994 .

[9]  Nando de Freitas,et al.  A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot , 2009, Auton. Robots.

[10]  Cheng Li,et al.  Filtering Bayesian optimization approach in weakly specified search space , 2018, Knowledge and Information Systems.

[11]  Jianxiang Xi,et al.  Robust time-varying formation design for multi-agent systems with disturbances: Extended-state-observer method , 2019, ArXiv.

[12]  Hamido Fujita,et al.  Robust Model Fitting Based on Greedy Search and Specified Inlier Threshold , 2019, IEEE Transactions on Industrial Electronics.

[13]  Gerald Penn,et al.  Convolutional Neural Networks for Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[14]  Hamido Fujita,et al.  Decision support system for arrhythmia prediction using convolutional neural network structure without preprocessing , 2019, Applied Intelligence.

[15]  Peter Auer,et al.  Using Confidence Bounds for Exploitation-Exploration Trade-offs , 2003, J. Mach. Learn. Res..

[16]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[17]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Sonja Georgievska,et al.  fMLC: fast multi-level clustering and visualization of large molecular datasets , 2018, Bioinform..

[19]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[20]  Andreas Krause,et al.  Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization , 2012, ICML.

[21]  Harold J. Kushner,et al.  A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise , 1964 .

[22]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[23]  David Ginsbourger,et al.  Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection , 2013, LION.

[24]  G. T. Timmer,et al.  Stochastic global optimization methods part I: Clustering methods , 1987, Math. Program..

[25]  Hao Wang,et al.  Cluster-based Kriging approximation algorithms for complexity reduction , 2017, Applied Intelligence.

[26]  András György,et al.  Efficient Multi-Start Strategies for Local Search Algorithms , 2009, J. Artif. Intell. Res..

[27]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[28]  Jonas Mockus,et al.  On Bayesian Methods for Seeking the Extremum , 1974, Optimization Techniques.

[29]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[30]  Qing-Long Han,et al.  Distributed Optimization for Multiagent Systems: An Edge-Based Fixed-Time Consensus Approach , 2019, IEEE Transactions on Cybernetics.

[31]  D. Ginsbourger,et al.  A benchmark of kriging-based infill criteria for noisy optimization , 2013, Structural and Multidisciplinary Optimization.

[32]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[33]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[34]  Hamido Fujita,et al.  Efficient Robust Model Fitting for Multistructure Data Using Global Greedy Search , 2020, IEEE Transactions on Cybernetics.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Shuzhi Sam Ge,et al.  Small traffic sign detection from large image , 2019, Applied Intelligence.

[37]  U. Rajendra Acharya,et al.  Application of deep convolutional neural network for automated detection of myocardial infarction using ECG signals , 2017, Inf. Sci..

[38]  Michael Muma,et al.  Gravitational Clustering: A Simple, Robust and Adaptive Approach for Distributed Networks , 2018, Signal Process..