Nonmyopic Gaussian Process Optimization with Macro-Actions

This paper presents a multi-staged approach to nonmyopic adaptive Gaussian process optimization (GPO) for Bayesian optimization (BO) of unknown, highly complex objective functions that, in contrast to existing nonmyopic adaptive BO algorithms, exploits the notion of macro-actions for scaling up to a further lookahead to match up to a larger available budget. To achieve this, we generalize GP upper confidence bound to a new acquisition function defined w.r.t. a nonmyopic adaptive macro-action policy, which is intractable to be optimized exactly due to an uncountable set of candidate outputs. The contribution of our work here is thus to derive a nonmyopic adaptive epsilon-Bayes-optimal macro-action GPO (epsilon-Macro-GPO) policy. To perform nonmyopic adaptive BO in real time, we then propose an asymptotically optimal anytime variant of our epsilon-Macro-GPO policy with a performance guarantee. We empirically evaluate the performance of our epsilon-Macro-GPO policy and its anytime variant in BO with synthetic and real-world datasets.

[1]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[2]  Michael A. Osborne,et al.  Gaussian Processes for Global Optimization , 2008 .

[3]  Kian Hsiang Low,et al.  Bayesian Optimization with Binary Auxiliary Information , 2019, UAI.

[4]  Adam D. Bull,et al.  Convergence Rates of Efficient Global Optimization Algorithms , 2011, J. Mach. Learn. Res..

[5]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[6]  Eric Walter,et al.  An informational approach to the global optimization of expensive-to-evaluate functions , 2006, J. Glob. Optim..

[7]  Kian Hsiang Low,et al.  Gaussian Process Decentralized Data Fusion and Active Sensing for Spatiotemporal Traffic Modeling and Prediction in Mobility-on-Demand Systems , 2015, IEEE Transactions on Automation Science and Engineering.

[8]  Zoubin Ghahramani,et al.  Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions , 2015, NIPS.

[9]  Kian Hsiang Low,et al.  Information-Theoretic Approach to Efficient Adaptive Path Planning for Mobile Robotic Environmental Sensing , 2009, ICAPS.

[10]  Nicholas Roy,et al.  PUMA: Planning Under Uncertainty with Macro-Actions , 2010, AAAI.

[11]  Kian Hsiang Low,et al.  Adaptive multi-robot wide-area exploration and mapping , 2008, AAMAS.

[12]  Doina Precup,et al.  Learning Options in Reinforcement Learning , 2002, SARA.

[13]  Mohan S. Kankanhalli,et al.  Active Learning Is Planning: Nonmyopic ε-Bayes-Optimal Active Learning of Gaussian Processes , 2014, ECML/PKDD.

[14]  Kian Hsiang Low,et al.  Multi-robot informative path planning for active sensing of environmental phenomena: a tale of two algorithms , 2013, AAMAS.

[15]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[16]  Kian Hsiang Low,et al.  Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression , 2019, AAAI.

[17]  Karen Willcox,et al.  Lookahead Bayesian Optimization with Inequality Constraints , 2017, NIPS.

[18]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[19]  Kian Hsiang Low,et al.  Active Markov information-theoretic path planning for robotic environmental sensing , 2011, AAMAS.

[20]  Kian Hsiang Low,et al.  GP-Localize: Persistent Mobile Robot Localization using Online Sparse Gaussian Process Observation Model , 2014, AAAI.

[21]  Karen Willcox,et al.  Bayesian Optimization with a Finite Budget: An Approximate Dynamic Programming Approach , 2016, NIPS.

[22]  Andreas Krause,et al.  Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization , 2012, ICML.

[23]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[24]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.

[25]  Neil D. Lawrence,et al.  Batch Bayesian Optimization via Local Penalization , 2015, AISTATS.

[26]  Kian Hsiang Low,et al.  Gaussian process decentralized data fusion meets transfer learning in large-scale distributed cooperative perception , 2017, Autonomous Robots.

[27]  Alan Fern,et al.  Batch Bayesian Optimization via Simulation Matching , 2010, NIPS.

[28]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[29]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations , 2013, UAI.

[30]  Peter I. Frazier,et al.  The Parallel Knowledge Gradient Method for Batch Bayesian Optimization , 2016, NIPS.

[31]  Kian Hsiang Low,et al.  Decentralized active robotic exploration and mapping for probabilistic field classification in environmental sensing , 2012, AAMAS.

[32]  Philipp Hennig,et al.  Entropy Search for Information-Efficient Global Optimization , 2011, J. Mach. Learn. Res..

[33]  Kian Hsiang Low,et al.  A Unifying Framework of Anytime Sparse Gaussian Process Regression Models with Stochastic Variational Inference for Big Data , 2015, ICML.

[34]  Gaurav S. Sukhatme,et al.  Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena , 2012, UAI.

[35]  Joelle Pineau,et al.  Bayesian reinforcement learning in continuous POMDPs with application to robot navigation , 2008, 2008 IEEE International Conference on Robotics and Automation.

[36]  David Hsu,et al.  Monte Carlo Value Iteration with Macro-Actions , 2011, NIPS.

[37]  Kian Hsiang Low,et al.  Gaussian Process Planning with Lipschitz Continuous Reward Functions: Towards Unifying Bayesian Optimization, Active Learning, and Beyond , 2015, AAAI.

[38]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[39]  Kian Hsiang Low,et al.  Generalized Online Sparse Gaussian Processes with Application to Persistent Mobile Robot Localization , 2014, ECML/PKDD.

[40]  David Ginsbourger,et al.  Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection , 2013, LION.

[41]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[42]  Kian Hsiang Low,et al.  Gaussian Process-Based Decentralized Data Fusion and Active Sensing for Mobility-on-Demand System , 2013, Robotics: Science and Systems.

[43]  Bryan Kian Hsiang Low,et al.  Information-Based Multi-Fidelity Bayesian Optimization , 2017 .

[44]  Kian Hsiang Low,et al.  Distributed Batch Gaussian Process Optimization , 2017, ICML.

[45]  Kian Hsiang Low,et al.  A Distributed Variational Inference Framework for Unifying Parallel Sparse Gaussian Process Regression Models , 2016, ICML.

[46]  Naomi Ehrich Leonard,et al.  Collective Motion, Sensor Networks, and Ocean Sampling , 2007, Proceedings of the IEEE.

[47]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[48]  E. Vázquez,et al.  Convergence properties of the expected improvement algorithm with fixed mean and covariance functions , 2007, 0712.3744.

[49]  Mohan S. Kankanhalli,et al.  Near-Optimal Active Learning of Multi-Output Gaussian Processes , 2015, AAAI.

[50]  Kian Hsiang Low,et al.  Multi-robot active sensing of non-stationary gaussian process-based environmental phenomena , 2014, AAMAS.

[51]  Nicolas Vayatis,et al.  Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration , 2013, ECML/PKDD.

[52]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[53]  Jesse Hoey,et al.  An analytic solution to discrete Bayesian reinforcement learning , 2006, ICML.

[54]  Kian Hsiang Low,et al.  Decentralized High-Dimensional Bayesian Optimization with Factor Graphs , 2017, AAAI.

[55]  Kian Hsiang Low,et al.  Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression , 2017, 2019 International Joint Conference on Neural Networks (IJCNN).

[56]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression for Big Data: Low-Rank Representation Meets Markov Approximation , 2014, AAAI.

[57]  Kian Hsiang Low,et al.  A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression , 2016, AAAI.

[58]  Neil D. Lawrence,et al.  GLASSES: Relieving The Myopia Of Bayesian Optimisation , 2015, AISTATS.

[59]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[60]  Stefan Zubrzycki,et al.  Lectures in probability theory and mathematical statistics , 1972 .

[61]  Francisco P. Chavez,et al.  Seawater sampling by an autonomous underwater vehicle: “Gulper” sample validation for nitrate, chlorophyll, phytoplankton, and primary production , 2016 .

[62]  Scott Sanner,et al.  Sequential Bayesian Optimisation for Spatial-Temporal Monitoring , 2014, UAI.

[63]  Kian Hsiang Low,et al.  Implicit Posterior Variational Inference for Deep Gaussian Processes , 2019, NeurIPS.

[64]  Kian Hsiang Low,et al.  Collective Online Learning of Gaussian Processes in Massive Multi-Agent Systems , 2019, AAAI.

[65]  Joon-Ho Choi,et al.  Post-occupancy evaluation of 20 office buildings as basis for future IEQ standards and guidelines , 2012 .

[66]  Kian Hsiang Low,et al.  Collective Model Fusion for Multiple Black-Box Experts , 2019, ICML.

[67]  Brahim Chaib-draa,et al.  Bayesian reinforcement learning in continuous POMDPs with gaussian processes , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  Kian Hsiang Low,et al.  Recent Advances in Scaling Up Gaussian Process Predictive Models for Large Spatiotemporal Data , 2014, DyDESS.

[69]  Nicholas Roy,et al.  Efficient Planning under Uncertainty with Macro-actions , 2014, J. Artif. Intell. Res..

[70]  Andrew G. Barto,et al.  Building Portable Options: Skill Transfer in Reinforcement Learning , 2007, IJCAI.