Optimization of hyperparameters of Gaussian process regression with the help of а low-order high-dimensional model representation: application to a potential energy surface

When the data are sparse, optimization of hyperparameters of the kernel in Gaussian process regression by the commonly used maximum likelihood estimation (MLE) criterion often leads to overfitting. We show that choosing hyperparameters based on a criterion of the completeness of the basis in the corresponding linear regression problem is superior to MLE. We show that this is facilitated by the use of High-dimensional model representation whereby a low-order HDMR representation can provide reliable reference functions and large synthetic test data sets needed for basis parameter optimization even with few data.

[1]  Matteo Fischetti,et al.  Embedded hyper-parameter tuning by Simulated Annealing , 2019, ArXiv.

[2]  Sergei Manzhos,et al.  Random Sampling High Dimensional Model Representation Gaussian Process Regression (RS-HDMR-GPR) for representing multidimensional functions with machine-learned lower-dimensional terms allowing insight with a general method , 2020, Comput. Phys. Commun..

[3]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[4]  Mohamed Ali Boussaidi,et al.  Random Sampling High Dimensional Model Representation Gaussian Process Regression (RS-HDMR-GPR) for Multivariate Function Representation: Application to Molecular Potential Energy Surfaces. , 2020, The journal of physical chemistry. A.

[5]  Koichi Yamashita,et al.  Fitting sparse multidimensional data with low-dimensional terms , 2009, Comput. Phys. Commun..

[6]  J. van Leeuwen,et al.  Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.

[7]  Sergei Manzhos,et al.  Neural network‐based approaches for building high dimensional and quantum dynamics‐friendly potential energy surfaces , 2015 .

[8]  T. Carrington,et al.  Neural Network Potential Energy Surfaces for Small Molecules and Reactions. , 2020, Chemical reviews.

[9]  S. Manzhos,et al.  Three-body interactions in clusters CO–(pH2)n , 2010 .

[10]  Aki Vehtari,et al.  An additive Gaussian process regression model for interpretable non-parametric analysis of longitudinal data , 2019, Nature Communications.

[11]  D. Chong.  Completeness profiles of one-electron basis sets , 1995 .

[12]  H. Rabitz,et al.  High Dimensional Model Representations , 2001 .

[13]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[14]  S Roberts,et al.  Gaussian processes for time-series modelling , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[15]  H. Nakai,et al.  Orbital-free density functional theory calculation applying semi-local machine-learned kinetic energy density functional and kinetic potential , 2020, Chemical Physics Letters.

[16]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[17]  Volker L. Deringer,et al.  Gaussian Process Regression for Materials and Molecules , 2021, Chemical reviews.

[18]  Aaron Klein,et al.  BOHB: Robust and Efficient Hyperparameter Optimization at Scale , 2018, ICML.

[19]  H. Rabitz,et al.  Random sampling-high dimensional model representation (RS-HDMR) and orthogonality of its different order component functions. , 2006, The journal of physical chemistry. A.

[20]  T. Carrington,et al.  Using an internal coordinate Gaussian basis and a space-fixed Cartesian coordinate kinetic energy operator to compute a vibrational spectrum with rectangular collocation. , 2016, The Journal of chemical physics.

[21]  Gregory Piatetsky-Shapiro,et al.  High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality , 2000 .

[22]  Sergei Manzhos,et al.  A model for the dissociative adsorption of N2O on Cu(1 0 0) using a continuous potential energy surface , 2010 .

[23]  Inverse Multiquadratic Functions as the Basis for the Rectangular Collocation Method to Solve the Vibrational Schrödinger Equation , 2018, Mathematics.

[24]  Sergei Manzhos,et al.  Easy representation of multivariate functions with low-dimensional terms via Gaussian process regression kernel design: applications to machine learning of potential energy surfaces and kinetic energy densities from sparse data , 2021, Mach. Learn. Sci. Technol..

[25]  Li Li,et al.  Understanding Machine-learned Density Functionals , 2014, ArXiv.

[26]  Hua Guo,et al.  Explicitly correlated MRCI-F12 potential energy surfaces for methane fit with several permutation invariant schemes and full-dimensional vibrational calculations , 2015 .

[27]  H. Rabitz,et al.  Practical Approaches To Construct RS-HDMR Component Functions , 2002 .

[28]  Sergei Manzhos,et al.  Kinetic energy densities based on the fourth order gradient expansion: performance in different classes of materials and improvement via machine learning. , 2018, Physical chemistry chemical physics : PCCP.

[29]  H. Nakai,et al.  Semi-local machine-learned kinetic energy density functional demonstrating smooth potential energy curves , 2019, Chemical Physics Letters.

[30]  Sergei Manzhos,et al.  A random-sampling high dimensional model representation neural network for building potential energy surfaces. , 2006, The Journal of chemical physics.

[31]  Simone A. Ludwig,et al.  Hyperparameter Optimization: Comparing Genetic Algorithm against Grid Search and Bayesian Optimization , 2021, 2021 IEEE Congress on Evolutionary Computation (CEC).

[32]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[33]  John C. Snyder,et al.  Orbital-free bond breaking via machine learning. , 2013, The Journal of chemical physics.

[34]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[35]  T. Carrington,et al.  A nested molecule-independent neural network approach for high-quality potential fits. , 2006, The journal of physical chemistry. A.

[37]  Carl E. Rasmussen,et al.  Additive Gaussian Processes , 2011, NIPS.

[38]  Ameet Talwalkar,et al.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..

[39]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[40]  T. Carrington,et al.  Computing the Anharmonic Vibrational Spectrum of UF6 in 15 Dimensions with an Optimized Basis Set and Rectangular Collocation. , 2015, The journal of physical chemistry. A.

[41]  H. Rabitz,et al.  General foundations of high‐dimensional model representations , 1999 .

[42]  Sergei Manzhos,et al.  Easy construction of representations of multivariate functions with low-dimensional terms via Gaussian process regression kernel design , 2021, ArXiv.

[43]  J. Bowman,et al.  An ab initio potential energy surface for the formic acid dimer: zero-point energy, selected anharmonic fundamental energies, and ground-state tunneling splitting calculated in relaxed 1-4-mode subspaces. , 2016, Physical chemistry chemical physics : PCCP.

[44]  Aditya Kamath,et al.  Neural networks vs Gaussian process regression for representing potential energy surfaces: A comparative study of fit quality and vibrational spectrum accuracy. , 2018, The Journal of chemical physics.

[45]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[46]  Yongdang Chen,et al.  Global sensitivity analysis of riveting parameters based on a random sampling-high dimensional model representation , 2021 .

[47]  T. Carrington,et al.  Extracting Functional Dependence from Sparse Data Using Dimensionality Reduction: Application to Potential Energy Surface Construction , 2011 .

[48]  J. Bowman,et al.  Permutationally invariant polynomial potential energy surfaces for tropolone and H and D atom tunneling dynamics. , 2020, The Journal of chemical physics.