Bayesian task embedding for few-shot Bayesian optimization

We describe a method for Bayesian optimization by which one may incorporate data from multiple systems whose quantitative interrelationships are unknown a priori. All general (nonreal-valued) features of the systems are associated with continuous latent variables that enter as inputs into a single metamodel that simultaneously learns the response surfaces of all of the systems. Bayesian inference is used to determine appropriate beliefs regarding the latent variables. We explain how the resulting probabilistic metamodel may be used for Bayesian optimization tasks and demonstrate its implementation on a variety of synthetic and real-world examples, comparing its performance under zero-, one-, and few-shot settings against traditional Bayesian optimization, which usually requires substantially more data from the system of interest.

[1]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[2]  Marc Peter Deisenroth,et al.  Doubly Stochastic Variational Inference for Deep Gaussian Processes , 2017, NIPS.

[3]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[4]  J. Sacks,et al.  Analysis of protein activity data by Gaussian stochastic process models. , 1999, Journal of biopharmaceutical statistics.

[5]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[6]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[7]  Wei Chen,et al.  Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables , 2019, Scientific Reports.

[8]  Noah D. Goodman,et al.  Pyro: Deep Universal Probabilistic Programming , 2018, J. Mach. Learn. Res..

[9]  James Hensman,et al.  Scalable Variational Gaussian Process Classification , 2014, AISTATS.

[10]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[11]  Ilias Bilionis,et al.  Multi-output local Gaussian process regression: Applications to uncertainty quantification , 2012, J. Comput. Phys..

[12]  Neil D. Lawrence,et al.  Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data , 2003, NIPS.

[13]  A. Kennedy,et al.  Hybrid Monte Carlo , 1988 .

[14]  Liping Wang,et al.  Finding Maximum Expected Improvement for High-Dimensional Design Optimization , 2019 .

[15]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[16]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[17]  Neil D. Lawrence,et al.  Bayesian Gaussian Process Latent Variable Model , 2010, AISTATS.

[18]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[19]  Kevin Ryan,et al.  Bayesian Multi-Source Modeling with Legacy Data , 2018 .

[20]  Nicholas Zabaras,et al.  Structured Bayesian Gaussian process latent variable model: applications to data-driven dimensionality reduction and high-dimensional inversion , 2019, J. Comput. Phys..

[21]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[22]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[23]  Kevin Ryan,et al.  Efficient robust design optimization using Gaussian process and intelligent sampling , 2018, 2018 Multidisciplinary Analysis and Optimization Conference.

[24]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[25]  Jasper Snoek,et al.  Multi-Task Bayesian Optimization , 2013, NIPS.

[26]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[27]  Wei Chen,et al.  Data-Centric Mixed-Variable Bayesian Optimization For Materials Design , 2019, Volume 2A: 45th Design Automation Conference.

[28]  Kevin Ryan,et al.  A Gaussian Process Modeling Approach for Fast Robust Design With Uncertain Inputs , 2018 .

[29]  Neil D. Lawrence,et al.  Semi-described and semi-supervised learning with Gaussian processes , 2015, UAI.

[30]  Ilias Bilionis,et al.  Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification , 2013, J. Comput. Phys..

[31]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[32]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[33]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[34]  Neil D. Lawrence,et al.  Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes , 2017, NIPS.

[35]  Andrew Gordon Wilson,et al.  Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) , 2015, ICML.

[36]  Stephen Tyree,et al.  Exact Gaussian Processes on a Million Data Points , 2019, NeurIPS.

[37]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[38]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[39]  Nicholas Zabaras,et al.  Structured Bayesian Gaussian process latent variable model , 2018, ArXiv.