A community-powered search of machine learning strategy space to find NMR property prediction models

The rise of machine learning (ML) has created an explosion in the potential strategies for using data to make scientific predictions. For physical scientists wishing to apply ML strategies to a particular domain, it can be difficult to assess in advance what strategy to adopt within a vast space of possibilities. Here we outline the results of an online community-powered effort to swarm search the space of ML strategies and develop algorithms for predicting atomic-pairwise nuclear magnetic resonance (NMR) properties in molecules. Using an open-source dataset, we worked with Kaggle to design and host a 3-month competition which received 47,800 ML model predictions from 2,700 teams in 84 countries. Within 3 weeks, the Kaggle community produced models with comparable accuracy to our best previously published "in-house" efforts. A meta-ensemble model constructed as a linear combination of the top predictions has a prediction accuracy which exceeds that of any individual model, 7-19x better than our previous state-of-the-art. The results highlight the potential of transformer architectures for predicting quantum mechanical (QM) molecular properties.

[1]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  I. Guyon,et al.  The Higgs Machine Learning Challenge , 2015 .

[3]  David Baker,et al.  Algorithm discovery by protein folding game players , 2011, Proceedings of the National Academy of Sciences.

[4]  Xavier Bresson,et al.  Residual Gated Graph ConvNets , 2017, ArXiv.

[5]  J. Pople,et al.  Self—Consistent Molecular Orbital Methods. XII. Further Extensions of Gaussian—Type Basis Sets for Use in Molecular Orbital Studies of Organic Molecules , 1972 .

[6]  Isabelle Guyon,et al.  The TrackML high-energy physics tracking challenge on Kaggle , 2019, EPJ Web of Conferences.

[7]  H. Sauermann,et al.  Crowd science user contribution patterns and their implications , 2015, Proceedings of the National Academy of Sciences.

[8]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[9]  J. Pople,et al.  Self‐consistent molecular orbital methods. XX. A basis set for correlated wave functions , 1980 .

[10]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[11]  Chi Chen,et al.  Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals , 2018, Chemistry of Materials.

[12]  Adrien Treuille,et al.  Predicting protein structures with a multiplayer online game , 2010, Nature.

[13]  Anita J. Brandolini.  NMRPredict Modgraph Consultants, Ltd, 1348 Graham Place, Escondido, CA 92129. http://www.modgraph-usa.com. Contact company for pricing information. , 2006 .

[14]  Heping Zhang,et al.  THE THEIL-SEN ESTIMATORS IN A MULTIPLE LINEAR REGRESSION MODEL , 2009 .

[15]  Alexandre Tkatchenko,et al.  Quantum-chemical insights from deep tensor neural networks , 2016, Nature Communications.

[16]  Michael J. Frisch,et al.  Self‐consistent molecular orbital methods 25. Supplementary functions for Gaussian basis sets , 1984 .

[17]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[18]  Lorenz C. Blum,et al.  970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. , 2009, Journal of the American Chemical Society.

[19]  Alexander Rich,et al.  Lessons for artificial intelligence from the study of natural stupidity , 2019, Nature Machine Intelligence.

[20]  Lingfan Yu,et al.  Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks. , 2019 .

[21]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[22]  Andrés M. Castillo,et al.  “Ask Ernö”: a self-learning tool for assignment and prediction of nuclear magnetic resonance spectra , 2016, Journal of Cheminformatics.

[23]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[24]  Danushka Bollegala,et al.  Joint Learning of Hierarchical Word Embeddings from a Corpus and a Taxonomy , 2019, AKBC.

[25]  Guy Lever,et al.  Human-level performance in 3D multiplayer games with population-based reinforcement learning , 2018, Science.

[26]  Balázs Kégl,et al.  The Higgs boson machine learning challenge , 2014, HEPML@NIPS.

[27]  David R. Glowacki,et al.  IMPRESSION – prediction of NMR parameters for 3-dimensional chemical structures using machine learning with near quantum chemical accuracy† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c9sc03854j , 2019, Chemical science.

[28]  Kevin Kaufmann,et al.  Crystal symmetry determination in electron diffraction using machine learning , 2020, Science.

[29]  J. Behler.  Atom-centered symmetry functions for constructing high-dimensional neural network potentials. , 2011, The Journal of chemical physics.

[30]  James Demmel,et al.  Large Batch Optimization for Deep Learning: Training BERT in 76 minutes , 2019, ICLR.

[31]  Frank Hutter,et al.  Fixing Weight Decay Regularization in Adam , 2017, ArXiv.

[32]  Risi Kondor,et al.  Cormorant: Covariant Molecular Neural Networks , 2019, NeurIPS.

[33]  Pieter P. Plehiers,et al.  A robotic platform for flow synthesis of organic compounds informed by AI planning , 2019, Science.

[34]  Cynthia Rudin,et al.  A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning , 2019, ArXiv.

[35]  Hao Wu,et al.  Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning , 2018, Science.

[36]  M. Hutson.  Artificial intelligence faces reproducibility crisis. , 2018, Science.

[37]  M. Frisch,et al.  Ab Initio Calculation of Vibrational Absorption and Circular Dichroism Spectra Using Density Functional Force Fields , 1994 .

[38]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[39]  Noam Brown,et al.  Superhuman AI for multiplayer poker , 2019, Science.

[40]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[41]  Jacob G Foster,et al.  Choosing experiments to accelerate collective discovery , 2015, Proceedings of the National Academy of Sciences.

[42]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[43]  Pavlo O. Dral,et al.  Quantum chemistry structures and properties of 134 kilo molecules , 2014, Scientific Data.

[44]  Chris Morley,et al.  Open Babel: An open chemical toolbox , 2011, J. Cheminformatics.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Michele Ceriotti,et al.  Chemical shifts in molecular solids by machine learning , 2018, Nature Communications.

[47]  K. Müller,et al.  Fast and accurate modeling of molecular atomization energies with machine learning. , 2011, Physical review letters.

[48]  J. Kruskal.  Nonmetric multidimensional scaling: A numerical method , 1964 .

[49]  Matthias Rupp,et al.  Big Data Meets Quantum Chemistry Approximations: The Δ-Machine Learning Approach. , 2015, Journal of chemical theory and computation.

[50]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[51]  Andrew Gordon Wilson,et al.  Averaging Weights Leads to Wider Optima and Better Generalization , 2018, UAI.

[52]  J. Kruskal.  Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis , 1964 .

[53]  Matthias Scheffler,et al.  Crowd-sourcing materials-science challenges with the NOMAD 2018 Kaggle competition , 2019, npj Computational Materials.

[54]  J. Pople,et al.  Self‐Consistent Molecular‐Orbital Methods. IX. An Extended Gaussian‐Type Basis for Molecular‐Orbital Studies of Organic Molecules , 1971 .

[55]  Alex Smola,et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs , 2019, ArXiv.

[56]  Anders S. Christensen,et al.  Alchemical and structural distribution based representation for universal quantum machine learning. , 2017, The Journal of chemical physics.

[57]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[58]  Nejc Dolensek,et al.  Facial expressions of emotion states and their neuronal correlates in mice , 2020, Science.

[59]  Anders S Christensen,et al.  FCHL revisited: Faster and more accurate quantum machine learning. , 2020, The Journal of chemical physics.

[60]  K. Müller,et al.  Machine Learning Predictions of Molecular Properties: Accurate Many-Body Potentials and Nonlocality in Chemical Space , 2015, The journal of physical chemistry letters.

[61]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[62]  Tommaso Calarco,et al.  Do physicists stop searches too early? A remote-science, optimization landscape investigation , 2017, ArXiv.

[63]  Florian Heigl,et al.  Opinion: Toward an international definition of citizen science , 2019, Proceedings of the National Academy of Sciences.

[64]  Patrick J. F. Groenen,et al.  Modern Multidimensional Scaling: Theory and Applications , 2003 .

[65]  James Demmel,et al.  Reducing BERT Pre-Training Time from 3 Days to 76 Minutes , 2019, ArXiv.

[66]  Michele Parrinello,et al.  Generalized neural-network representation of high-dimensional potential-energy surfaces. , 2007, Physical review letters.

[67]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[68]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.