Bayesian Optimization for Adaptive Experimental Design: A Review

Bayesian optimisation is a statistical method that efficiently models and optimises expensive “black-box” functions. This review considers the application of Bayesian optimisation to experimental design, in comparison to existing Design of Experiments (DOE) methods. Solutions are surveyed for a range of core issues in experimental design including: the incorporation of prior knowledge, high dimensional optimisation, constraints, batch evaluation, multiple objectives, multi-fidelity data, and mixed variable types.

[1]  Harold J. Kushner,et al.  A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise , 1964 .

[2]  Karen Willcox,et al.  Lookahead Bayesian Optimization with Inequality Constraints , 2017, NIPS.

[3]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[4]  Peter I. Frazier,et al.  Bayesian optimization for materials design , 2015, 1506.01349.

[5]  Michèle Sebag,et al.  Collaborative hyperparameter tuning , 2013, ICML.

[6]  James Theiler,et al.  Accelerated search for materials with targeted properties by adaptive design , 2016, Nature Communications.

[7]  Svetha Venkatesh,et al.  New Bayesian-Optimization-Based Design of High-Strength 7xxx-Series Alloys from Recycled Aluminum , 2018, JOM.

[8]  Christine M. Anderson-Cook,et al.  Process Optimization for Multiple Responses Utilizing the Pareto Front Approach , 2014 .

[9]  Victor Picheny,et al.  Multiobjective optimization using Gaussian process emulators via stepwise uncertainty reduction , 2013, Statistics and Computing.

[10]  Gerald Tesauro,et al.  Selecting Near-Optimal Learners via Incremental Data Allocation , 2015, AAAI.

[11]  G. Box,et al.  On the Experimental Attainment of Optimum Conditions , 1951 .

[12]  Aki Vehtari,et al.  Gaussian processes with monotonicity information , 2010, AISTATS.

[13]  Gideon S. Mann,et al.  Efficient Transfer Learning Method for Automatic Hyperparameter Tuning , 2014, AISTATS.

[14]  Svetha Venkatesh,et al.  Exploiting Strategy-Space Diversity for Batch Bayesian Optimization , 2018, AISTATS.

[15]  Turab Lookman,et al.  Multi-objective Optimization for Materials Discovery via Adaptive Design , 2018, Scientific Reports.

[16]  F. Viana.  Things you wanted to know about the Latin hypercube design and were afraid to ask , 2016 .

[17]  Cheng Li,et al.  Budgeted Batch Bayesian Optimization , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[18]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[19]  Ida Gremyr,et al.  A review of practices for robust design methodology , 2009 .

[20]  Aki Vehtari,et al.  Bayesian Optimization of Unimodal Functions , 2017 .

[21]  Karl Heinz Kienitz,et al.  Efficient, nearly orthogonal-and-balanced, mixed designs: an effective way to conduct trade-off analyses via simulation , 2013, J. Simulation.

[22]  Carl E. Rasmussen,et al.  Additive Gaussian Processes , 2011, NIPS.

[23]  Ruben Martinez-Cantin,et al.  BayesOpt: a Bayesian optimization library for nonlinear optimization, experimental design and bandits , 2014, J. Mach. Learn. Res..

[24]  Wolfgang Ponweiser,et al.  Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted -Metric Selection , 2008, PPSN.

[25]  Alan Fern,et al.  Batch Bayesian Optimization via Simulation Matching , 2010, NIPS.

[26]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[27]  Cheng Li,et al.  High Dimensional Bayesian Optimization using Dropout , 2018, IJCAI.

[28]  P. Siarry,et al.  Multiobjective Optimization: Principles and Case Studies , 2004 .

[29]  Shin Kiyohara,et al.  Bayesian optimization for efficient determination of metal oxide grain boundary structures , 2017 .

[30]  Kamalika Chaudhuri,et al.  Active Learning from Weak and Strong Labelers , 2015, NIPS.

[31]  G. Karniadakis,et al.  Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond , 2016, Journal of The Royal Society Interface.

[32]  Jan Peters,et al.  Bayesian optimization for learning gaits under uncertainty , 2015, Annals of Mathematics and Artificial Intelligence.

[33]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[34]  Junichiro Shiomi,et al.  Designing Nanostructures for Phonon Transport via Bayesian Optimization , 2016, 1609.04972.

[35]  V'ictor Pena,et al.  Bayesian Optimization with Shape Constraints , 2016, 1612.08915.

[36]  Zi Wang,et al.  Batched High-dimensional Bayesian Optimization via Structural Kernel Learning , 2017, ICML.

[37]  Christoforos Anagnostopoulos,et al.  The Automatic Neuroscientist: A framework for optimizing experimental design with closed-loop real-time fMRI , 2016, NeuroImage.

[38]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[39]  Matthias Poloczek,et al.  Multi-Information Source Optimization , 2016, NIPS.

[40]  Peter J. Lenk,et al.  Bayesian analysis of shape-restricted functions using Gaussian process priors , 2017 .

[41]  Peter I. Frazier,et al.  The Parallel Knowledge Gradient Method for Batch Bayesian Optimization , 2016, NIPS.

[42]  Svetha Venkatesh,et al.  Multi-objective Bayesian optimisation with preferences over objectives , 2019, NeurIPS.

[43]  Victor Picheny,et al.  Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian , 2016, NIPS.

[44]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Bayesian Optimization with Unknown Constraints , 2015, ICML.

[45]  Neil D. Lawrence,et al.  Batch Bayesian Optimization via Local Penalization , 2015, AISTATS.

[46]  Antoine Cully,et al.  Robots that can adapt like animals , 2014, Nature.

[47]  Max Welling,et al.  BOCK : Bayesian Optimization with Cylindrical Kernels , 2018, ICML.

[48]  Jian Peng,et al.  Quantile Stein Variational Gradient Descent for Batch Bayesian Optimization , 2019, ICML.

[49]  Joshua D. Knowles,et al.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems , 2006, IEEE Transactions on Evolutionary Computation.

[50]  Atsuto Seko,et al.  Prediction of Low-Thermal-Conductivity Compounds with First-Principles Anharmonic Lattice-Dynamics Calculations and Bayesian Optimization. , 2015, Physical review letters.

[51]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[52]  Iftekhar A. Karimi,et al.  Design of computer experiments: A review , 2017, Comput. Chem. Eng..

[53]  Andreas Krause,et al.  Bayesian optimization for maximum power point tracking in photovoltaic power plants , 2016, 2016 European Control Conference (ECC).

[54]  Eduardo C. Garrido-Merchán,et al.  Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes , 2017, Neurocomputing.

[55]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[56]  J. Hogden,et al.  Statistical inference and adaptive design for materials discovery , 2017 .

[57]  D. Ginsbourger,et al.  A Multi-points Criterion for Deterministic Parallel Global Optimization based on Gaussian Processes , 2008 .

[58]  F. Hutter,et al.  Towards efficient Bayesian Optimization for Big Data , 2015 .

[59]  Jennifer G. Dy,et al.  ADMMBO: Bayesian Optimization with Unknown Constraints using ADMM , 2019, J. Mach. Learn. Res..

[60]  M. Emmerich,et al.  The computation of the expected improvement in dominated hypervolume of Pareto front approximations , 2008 .

[61]  Svetha Venkatesh,et al.  Information-Theoretic Transfer Learning Framework for Bayesian Optimisation , 2018, ECML/PKDD.

[62]  Yun-En Liu,et al.  Designing Engaging Games Using Bayesian Optimization , 2016, CHI.

[63]  Ryan P. Adams,et al.  Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach. , 2016, Nature materials.

[64]  J. I.  The Design of Experiments , 1936, Nature.

[65]  Susan M. Lewis,et al.  Design of experiments for screening , 2015, 1510.05248.

[66]  Roman Garnett,et al.  Discovering and Exploiting Additive Structure for Bayesian Optimization , 2017, AISTATS.

[67]  Atsuto Seko,et al.  Machine learning with systematic density-functional theory calculations: Application to melting temperatures of single- and binary-component solids , 2013, 1310.1546.

[68]  Peter I. Frazier,et al.  Parallel Bayesian Global Optimization of Expensive Functions , 2016, Oper. Res..

[69]  田口 玄一,et al.  Introduction to quality engineering : designing quality into products and processes , 1986 .

[70]  Nando de Freitas,et al.  Bayesian Optimization in High Dimensions via Random Embeddings , 2013, IJCAI.

[71]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[72]  Andreas Krause,et al.  Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization , 2012, ICML.

[73]  Peter I. Frazier,et al.  A Tutorial on Bayesian Optimization , 2018, ArXiv.

[74]  Matthias W. Seeger,et al.  Bayesian Optimization with Tree-structured Dependencies , 2017, ICML.

[75]  Andreas Krause,et al.  Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with Bayesian optimization , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[76]  Svetha Venkatesh,et al.  Regret Bounds for Transfer Learning in Bayesian Optimisation , 2017, AISTATS.

[77]  Sabine Schulze.  Screening Methods For Experimentation In Industry Drug Discovery And Genetics , 2016 .

[78]  Simon Moritz Göhler,et al.  A Framework for the Application of Robust Design Methods and Tools , 2014 .

[79]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[80]  S. Rana,et al.  Bayesian Optimization with Monotonicity Information , 2017 .

[81]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[82]  Andreas Krause,et al.  Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces , 2019, ICML.

[83]  Daniel Hern'andez-Lobato,et al.  Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints , 2016, Neurocomputing.

[84]  Julien Bect,et al.  A Bayesian approach to constrained single- and multi-objective optimization , 2015, Journal of Global Optimization.

[85]  Andreas Krause,et al.  Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting , 2009, IEEE Transactions on Information Theory.

[86]  Jenessa Lancaster,et al.  Bayesian Optimization for Neuroimaging Pre-processing in Brain Age Classification and Prediction , 2018, Front. Aging Neurosci..

[87]  Matt J. Kusner,et al.  Bayesian Optimization with Inequality Constraints , 2014, ICML.

[88]  Jasper Snoek,et al.  Bayesian Optimization with Unknown Constraints , 2014, UAI.

[89]  P Baldi,et al.  Enhanced Higgs boson to τ(+)τ(-) search with deep learning. , 2014, Physical review letters.

[90]  Kirthevasan Kandasamy,et al.  Multi-fidelity Bayesian Optimisation with Continuous Approximations , 2017, ICML.

[91]  Jasper Snoek,et al.  Multi-Task Bayesian Optimization , 2013, NIPS.

[92]  Michael A. Osborne,et al.  A Kernel for Hierarchical Parameter Spaces , 2013, ArXiv.

[93]  Svetha Venkatesh,et al.  Process-constrained batch Bayesian optimisation , 2017, NIPS.

[94]  Andreas Krause,et al.  High-Dimensional Gaussian Process Bandits , 2013, NIPS.

[95]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[96]  Xin Yao,et al.  Stochastic ranking for constrained evolutionary optimization , 2000, IEEE Trans. Evol. Comput..

[97]  Villegas García,et al.  An investigation into new kernels for categorical variables , 2013 .

[98]  Cheng Li,et al.  Rapid Bayesian optimisation for synthesis of short polymer fiber materials , 2017, Scientific Reports.

[99]  Chun-Liang Li,et al.  High Dimensional Bayesian Optimization via Restricted Projection Pursuit Models , 2016, AISTATS.

[100]  Svetha Venkatesh,et al.  Batch Bayesian optimization using multi-scale search , 2020, Knowl. Based Syst..

[101]  Andreas Krause,et al.  Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features , 2018, NeurIPS.

[102]  Svetha Venkatesh,et al.  Flexible Transfer Learning Framework for Bayesian Optimisation , 2016, PAKDD.

[103]  Ricardo Pio Monti,et al.  Dissociating frontoparietal brain networks with neuroadaptive Bayesian optimization , 2017, Nature Communications.

[104]  Kirthevasan Kandasamy,et al.  High Dimensional Bayesian Optimisation and Bandits via Additive Models , 2015, ICML.

[105]  Cheng Li,et al.  High Dimensional Bayesian Optimization with Elastic Gaussian Process , 2017, ICML.

[106]  Zoubin Ghahramani,et al.  Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions , 2015, NIPS.

[107]  M. Powell.  A View of Algorithms for Optimization without Derivatives 1 , 2007 .

[108]  Matthias Poloczek,et al.  A Framework for Bayesian Optimization in Embedded Subspaces , 2019, ICML.

[109]  Haitao Liu,et al.  A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design , 2017, Structural and Multidisciplinary Optimization.

[110]  Daniel M. Packwood,et al.  Bayesian Optimization for Materials Science , 2017 .

[111]  Koji Tsuda,et al.  COMBO: An efficient Bayesian optimization library for materials science , 2016 .

[112]  G. Gary Wang,et al.  Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions , 2010 .

[113]  Connie M. Borror,et al.  Response Surface Methodology: A Retrospective and Literature Survey , 2004 .

[114]  Michael A. Osborne,et al.  Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces , 2014, 1409.4011.

[115]  Nicolas Vayatis,et al.  Parallel Gaussian Process Optimization with Upper Confidence Bound and Pure Exploration , 2013, ECML/PKDD.

[116]  G. Gary Wang,et al.  Review of Metamodeling Techniques in Support of Engineering Design Optimization , 2007, DAC 2006.