HOAX: a hyperparameter optimisation algorithm explorer for neural networks

Computational chemistry has become an important tool to predict and understand molecular properties and reactions. Even though recent years have seen a significant growth in new algorithms and computational methods that speed up quantum chemical calculations, the bottleneck for trajectory-based methods to study photoinduced processes is still the huge number of electronic structure calculations. In this work, we present an innovative solution, in which the amount of electronic structure calculations is drastically reduced, by employing machine learning algorithms and methods borrowed from the realm of artificial intelligence. However, applying these algorithms effectively requires finding optimal hyperparameters, which remains a challenge itself. Here we present an automated user-friendly framework, HOAX, to perform the hyperparameter optimization for neural networks, which bypasses the need for a lengthy manual process. The neural network generated potential energy surfaces (PESs) reduces the computational costs compared to the ab initio-based PESs. We perform a comparative investigation on the performance of different hyperparameter optimiziation algorithms, namely grid search, simulated annealing, genetic algorithm, and bayesian optimizer in finding the optimal hyperparameters necessary for constructing the well-performing neural network in order to fit the PESs of small organic molecules. Our results show that this automated toolkit not only facilitate a straightforward way to perform the hyperparameter optimization but also the resulting neural networks-based generated PESs are in reasonable agreement with the ab initio-based PESs.

[1]  Kristof T. Schütt,et al.  Roadmap on Machine learning in electronic structure , 2022, Electronic Structure.

[2]  Daniel S. Levine,et al.  Software for the frontiers of quantum chemistry: An overview of developments in the Q-Chem 5 package , 2021, The Journal of chemical physics.

[3]  B. Ensing,et al.  Discovering Collective Variables of Molecular Transitions via Genetic Algorithms and Neural Networks , 2021, Journal of chemical theory and computation.

[4]  M. Menger,et al.  PySurf: A Framework for Database Accelerated Direct Dynamics , 2020, Journal of chemical theory and computation.

[5]  P. Marquetand,et al.  Machine Learning for Electronically Excited States of Molecules , 2020, Chemical reviews.

[6]  Julia Westermayr,et al.  Combining SchNet and SHARC: The SchNarc Machine Learning Approach for Excited-State Dynamics , 2020, The journal of physical chemistry letters.

[7]  D. Yarkony,et al.  Accurate Neural Network Representation of the ab initio Determined Spin-Orbit Interaction in the Diabatic Representation including the Effects of Conical Intersections. , 2020, The journal of physical chemistry letters.

[8]  Zhecan Wang,et al.  Learning Visual Commonsense for Robust Scene Graph Generation , 2020, ECCV.

[9]  F. Noé,et al.  Deep-neural-network solution of the electronic Schrödinger equation , 2019, Nature Chemistry.

[10]  V. B. Surya Prasath,et al.  Choosing Mutation and Crossover Ratios for Genetic Algorithms - A Review with a New Dynamic Approach , 2019, Inf..

[11]  J. Bowman,et al.  Two-layer Gaussian-based MCTDH study of the S1 ← S0 vibronic absorption spectrum of formaldehyde using multiplicative neural network potentials , 2019, The Journal of Chemical Physics.

[12]  Ove Christiansen,et al.  Machine learning for potential energy surfaces: An extensive database and assessment of methods. , 2019, The Journal of chemical physics.

[13]  Daniel G A Smith,et al.  PES-Learn: An Open-Source Software Package for the Automated Generation of Machine Learning Models of Molecular Potential Energy Surfaces. , 2019, Journal of chemical theory and computation.

[14]  Anders S. Christensen,et al.  Operators in quantum machine learning: Response properties in chemical space. , 2018, The Journal of chemical physics.

[15]  Robert F. Stengel,et al.  An introduction to neural networks , 2018 .

[16]  Walter Thiel,et al.  Nonadiabatic Excited-State Dynamics with Machine Learning , 2018, The journal of physical chemistry letters.

[17]  Anders S. Christensen,et al.  Alchemical and structural distribution based representation for universal quantum machine learning. , 2017, The Journal of chemical physics.

[18]  K-R Müller,et al.  SchNet - A deep learning architecture for molecules and materials. , 2017, The Journal of chemical physics.

[19]  Justin S. Smith,et al.  Hierarchical modeling of molecular energies using a deep neural network. , 2017, The Journal of chemical physics.

[20]  Konstantin Gubaev,et al.  Machine learning of molecular properties: Locality and active learning. , 2017, The Journal of chemical physics.

[21]  Jessica Baldwin-Philippi.  The Myths of Data-Driven Campaigning , 2017 .

[22]  Alexandre Tkatchenko,et al.  Quantum-chemical insights from deep tensor neural networks , 2016, Nature Communications.

[23]  Chris Eliasmith,et al.  Hyperopt: a Python library for model selection and hyperparameter optimization , 2015 .

[24]  M. Rupp,et al.  Machine Learning for Quantum Mechanical Properties of Atoms in Molecules , 2015, 1505.00350.

[25]  Raghunathan Ramakrishnan,et al.  Many Molecular Properties from One Kernel in Chemical Space. , 2015, Chimia.

[26]  Pavlo O. Dral,et al.  Quantum chemistry structures and properties of 134 kilo molecules , 2014, Scientific Data.

[27]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[28]  Shifei Ding,et al.  An optimizing BP neural network algorithm based on genetic algorithm , 2011, Artificial Intelligence Review.

[29]  W. Hase,et al.  Comparisons of classical and Wigner sampling of transition state energy levels for quasiclassical trajectory chemical dynamics simulations. , 2010, The Journal of chemical physics.

[30]  Sanjeev R. Kulkarni,et al.  Reliable Reasoning: Induction and Statistical Learning Theory , 2007 .

[31]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[32]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[33]  Anna I. Krylov,et al.  The spin–flip approach within time-dependent density functional theory: Theory and applications to diradicals , 2003 .

[34]  H. J. Mclaughlin,et al.  Learn , 2002 .

[35]  G. Worth,et al.  Molecular dynamics of pyrazine after excitation to the S2 electronic state using a realistic 24-mode model Hamiltonian , 1999 .

[36]  Paul F. Dubois,et al.  Software for Portable Scientific Data Management , 1993 .

[37]  A. Becke.  A New Mixing of Hartree-Fock and Local Density-Functional Theories , 1993 .

[38]  Russ Rew,et al.  NetCDF: an interface for scientific data access , 1990, IEEE Computer Graphics and Applications.

[39]  T. H. Dunning.  Gaussian basis sets for use in correlated molecular calculations. I. The atoms boron through neon and hydrogen , 1989 .

[40]  Lawrence Davis,et al.  Genetic Algorithms and Simulated Annealing , 1987 .

[41]  E. Gardner.  Structure of metastable states in the Hopfield model , 1986 .

[42]  J. Pople,et al.  Self—Consistent Molecular Orbital Methods. XII. Further Extensions of Gaussian—Type Basis Sets for Use in Molecular Orbital Studies of Organic Molecules , 1972 .

[43]  A. Gray,et al.  I. THE ORIGIN OF SPECIES BY MEANS OF NATURAL SELECTION , 1963 .

[44]  A. Bennett.  The Origin of Species by means of Natural Selection; or the Preservation of Favoured Races in the Struggle for Life , 1872, Nature.