Deep learning for computational chemistry

The rise and fall of artificial neural networks is well documented in the scientific literature of both computer science and computational chemistry. Yet almost two decades later, we are now seeing a resurgence of interest in deep learning, a machine learning algorithm based on multilayer neural networks. Within the last few years, we have seen the transformative impact of deep learning in many domains, particularly in speech recognition and computer vision, to the extent that the majority of expert practitioners in those field are now regularly eschewing prior established models in favor of deep learning models. In this review, we provide an introductory overview into the theory of deep neural networks and their unique properties that distinguish them from traditional machine learning algorithms used in cheminformatics. By providing an overview of the variety of emerging applications of deep neural networks, we highlight its ubiquity and broad applicability to a wide range of challenges in the field, including quantitative structure activity relationship, virtual screening, protein structure prediction, quantum chemistry, materials design, and property prediction. In reviewing the performance of deep neural networks, we observed a consistent outperformance against non‐neural networks state‐of‐the‐art models across disparate research topics, and deep neural network‐based models often exceeded the “glass ceiling” expectations of their respective tasks. Coupled with the maturity of GPU‐accelerated computing for training deep neural networks and the exponential growth of chemical data on which to train these networks on, we anticipate that deep learning algorithms will be a valuable tool for computational chemistry. © 2017 Wiley Periodicals, Inc.

[1]  Geoffrey Zweig,et al.  Achieving Human Parity in Conversational Speech Recognition , 2016, ArXiv.

[2]  Sanjay Joshua Swamidass,et al.  A simple model predicts UGT-mediated metabolism , 2016, Bioinform..

[3]  Quoc V. Le,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[4]  G. Barta Identifying Biological Pathway Interrupting Toxins Using Multi-Tree Ensembles , 2016, Front. Environ. Sci..

[5]  S. Joshua Swamidass,et al.  Modeling Reactivity to Biological Macromolecules with a Deep Multitask Network , 2016, ACS central science.

[6]  Brian Goldman,et al.  Modeling Industrial ADMET Data with Multitask Networks , 2016, 1606.08793.

[7]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[8]  Carlos Guestrin,et al.  Model-Agnostic Interpretability of Machine Learning , 2016, ArXiv.

[9]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[10]  John Salvatier,et al.  Theano: A Python framework for fast computation of mathematical expressions , 2016, ArXiv.

[11]  Paul Raccuglia,et al.  Machine-learning-assisted materials discovery using failed experiments , 2016, Nature.

[12]  Alex Zhavoronkov,et al.  Applications of Deep Learning in Biomedicine. , 2016, Molecular pharmaceutics.

[13]  Igor V. Tetko,et al.  Consensus Modeling for HTS Assays Using In silico Descriptors Calculates the Best Balanced Accuracy in Tox21 Challenge , 2016, Front. Environ. Sci..

[14]  Günter Klambauer,et al.  DeepTox: Toxicity Prediction using Deep Learning , 2016, Front. Environ. Sci..

[15]  Chris J. Maddison,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[16]  G. Schneider,et al.  Deep Learning in Drug Discovery , 2016, Molecular informatics.

[17]  Pierre Baldi,et al.  Accurate and efficient target prediction using a potency-sensitive influence-relevance voter , 2015, Journal of Cheminformatics.

[18]  Edward O. Pyzer-Knapp,et al.  Learning from the Harvard Clean Energy Project: The Use of Neural Networks to Accelerate Materials Discovery , 2015 .

[19]  B. Alsberg,et al.  Artificial evolution of coumarin dyes for dye sensitized solar cells. , 2015, Physical chemistry chemical physics : PCCP.

[20]  Jianyang Zeng,et al.  A deep learning framework for modeling structural features of RNA-binding protein targets , 2015, Nucleic acids research.

[21]  Luhua Lai,et al.  Deep Learning for Drug-Induced Liver Injury , 2015, J. Chem. Inf. Model..

[22]  Izhar Wallach,et al.  AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery , 2015, ArXiv.

[23]  Sergei V. Kalinin,et al.  Big-deep-smart data in imaging for guiding materials design. , 2015, Nature materials.

[24]  B. Frey,et al.  Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning , 2015, Nature Biotechnology.

[25]  James G. Lyons,et al.  Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by iterative deep learning , 2015, Scientific Reports.

[26]  S. Joshua Swamidass,et al.  Modeling Epoxidation of Drug-like Molecules with a Deep Machine Learning Network , 2015, ACS central science.

[27]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[28]  Sanjay Joshua Swamidass,et al.  Improved Prediction of CYP-Mediated Metabolism with Chemical Fingerprints , 2015, J. Chem. Inf. Model..

[29]  David M. Eike,et al.  Accurate modeling of ionic surfactants at high concentration. , 2015, The journal of physical chemistry. B.

[30]  Walter Thiel,et al.  Machine Learning of Parameters for Accurate Semiempirical Quantum Chemical Calculations , 2015, Journal of chemical theory and computation.

[31]  Felix A Faber,et al.  Crystal structure representations for machine learning models of formation energies , 2015, 1503.07406.

[32]  Pavlo O. Dral,et al.  Big Data Meets Quantum Chemistry Approximations: The Δ-Machine Learning Approach. , 2015, Journal of chemical theory and computation.

[33]  S. Joshua Swamidass,et al.  Site of reactivity models predict molecular reactivity of diverse chemicals with glutathione. , 2015, Chemical research in toxicology.

[34]  Robert P. Sheridan,et al.  Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships , 2015, J. Chem. Inf. Model..

[35]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[36]  Vijay S. Pande,et al.  Massively Multitask Networks for Drug Discovery , 2015, ArXiv.

[37]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Kuldip K. Paliwal,et al.  Predicting backbone Cα angles and dihedrals from protein sequences by stacked sparse auto‐encoder deep neural network , 2014, J. Comput. Chem..

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[41]  David M. Reif,et al.  Profiling of the Tox21 10K compound library for agonists and antagonists of the estrogen receptor alpha signaling pathway , 2014, Scientific Reports.

[42]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[43]  Navdeep Jaitly,et al.  Multi-task Neural Networks for QSAR Predictions , 2014, ArXiv.

[44]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[45]  P. Baldi,et al.  Searching for exotic particles in high-energy physics with deep learning , 2014, Nature Communications.

[46]  O. Anatole von Lilienfeld,et al.  Modeling electronic quantum transport with machine learning , 2014, 1401.8277.

[47]  Yang Li,et al.  Stalking the Materials Genome: A Data‐Driven Approach to the Virtual Design of Nanostructured Polymers , 2013, Advanced functional materials.

[48]  S. Joshua Swamidass,et al.  XenoSite: Accurately Predicting CYP-Mediated Sites of Metabolism with Neural Networks , 2013, J. Chem. Inf. Model..

[49]  Jing Huang,et al.  CHARMM36 all‐atom additive protein force field: Validation based on comparison to NMR data , 2013, J. Comput. Chem..

[50]  Frederick R. Manby,et al.  Machine-learning approach for one- and two-body corrections to density functional theory: Applications to molecular and condensed water , 2013 .

[51]  Klaus-Robert Müller,et al.  Assessment and Validation of Machine Learning Methods for Predicting Molecular Atomization Energies. , 2013, Journal of chemical theory and computation.

[52]  Kristin A. Persson,et al.  Commentary: The Materials Project: A materials genome approach to accelerating materials innovation , 2013 .

[53]  M. Rupp,et al.  Fourier series of atomic radial distribution functions: A molecular fingerprint for machine learning models of quantum chemical properties , 2013, 1307.2918.

[54]  Pierre Baldi,et al.  Deep Architectures and Deep Learning in Chemoinformatics: The Prediction of Aqueous Solubility for Drug-Like Molecules , 2013, J. Chem. Inf. Model..

[55]  M. Rupp,et al.  Machine learning of molecular electronic properties in chemical compound space , 2013, 1305.7074.

[56]  David Ryan Koes,et al.  Lessons Learned in Empirical Scoring with smina from the CSAR 2011 Benchmarking Exercise , 2013, J. Chem. Inf. Model..

[57]  Atina G. Coté,et al.  Evaluation of methods for modeling transcription factor sequence specificity , 2013, Nature Biotechnology.

[58]  Michael P Snyder,et al.  High-throughput sequencing for biology and medicine , 2013, Molecular systems biology.

[59]  Jianlin Cheng,et al.  Predicting protein residue-residue contacts using deep networks and boosting , 2012, Bioinform..

[60]  Thomas A. Hopf,et al.  Protein structure prediction from sequence variation , 2012, Nature Biotechnology.

[61]  Pierre Baldi,et al.  Deep architectures for protein contact map prediction , 2012, Bioinform..

[62]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Michael M. Mysinger,et al.  Directory of Useful Decoys, Enhanced (DUD-E): Better Ligands and Decoys for Better Benchmarking , 2012, Journal of medicinal chemistry.

[64]  J. Reymond,et al.  Exploring chemical space for drug discovery using the chemical universe database. , 2012, ACS chemical neuroscience.

[65]  Rhiju Das,et al.  Are Protein Force Fields Getting Better? A Systematic Benchmark on 524 Diverse NMR Measurements. , 2012, Journal of chemical theory and computation.

[66]  Yaoqi Zhou,et al.  SPINE X: Improving protein secondary structure prediction by multistep learning coupled with prediction of solvent accessible surface area and backbone torsion angles , 2012, J. Comput. Chem..

[67]  Frank R Burden,et al.  Quantitative structure-property relationship modeling of diverse materials properties. , 2012, Chemical reviews.

[68]  Evan Bolton,et al.  PubChem's BioAssay Database , 2011, Nucleic Acids Res..

[69]  Jianwen Fang,et al.  Predicting residue-residue contacts using random forest models , 2011, Bioinform..

[70]  R. Dror,et al.  How Fast-Folding Proteins Fold , 2011, Science.

[71]  K. Müller,et al.  Fast and accurate modeling of molecular atomization energies with machine learning. , 2011, Physical review letters.

[72]  CHUN WEI YAP,et al.  PaDEL‐descriptor: An open source software to calculate molecular descriptors and fingerprints , 2011, J. Comput. Chem..

[73]  Paul E. Smith,et al.  A Kirkwood-Buff Derived Force Field for Aqueous Alkali Halides. , 2011, Journal of chemical theory and computation.

[74]  Derek C. Rose,et al.  Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[75]  A. Tropsha,et al.  Quantitative nanostructure-activity relationship modeling. , 2010, ACS nano.

[76]  Garrett B. Goh,et al.  Quantitative Structure‐Fluorescence Property Relationship Analysis of a Large BODIPY Library , 2010, Molecular informatics.

[77]  Tjerk P. Straatsma,et al.  NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations , 2010, Comput. Phys. Commun..

[78]  Alexander Tropsha,et al.  Best Practices for QSAR Model Development, Validation, and Exploitation , 2010, Molecular informatics.

[79]  Anubhav Jain,et al.  Finding Nature’s Missing Ternary Oxide Compounds Using Machine Learning and Density Functional Theory , 2010 .

[80]  M. Hahn,et al.  Extended-Connectivity Fingerprints , 2010, J. Chem. Inf. Model..

[81]  Jitender Verma,et al.  3D-QSAR in drug design--a review. , 2010, Current topics in medicinal chemistry.

[82]  Yuedong Yang,et al.  Predicting continuous local structure and the effect of its substitution for secondary structure in fragment-free protein structure prediction. , 2009, Structure.

[83]  Jianpeng Ma,et al.  CHARMM: The biomolecular simulation program , 2009, J. Comput. Chem..

[84]  A. Olson,et al.  AutoDock Vina: Improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading , 2009, J. Comput. Chem..

[85]  V. Navarro,et al.  Human drug hepatotoxicity: a contemporary clinical perspective , 2009 .

[86]  Torgeir R. Hvidsten,et al.  Using multi-data hidden Markov models trained on local neighborhoods of protein structure to predict residue-residue contacts , 2009, Bioinform..

[87]  S. Joshua Swamidass,et al.  Influence Relevance Voting: An Accurate And Interpretable Virtual High Throughput Screening Method , 2009, J. Chem. Inf. Model..

[88]  Qing-Song Xu,et al.  Support vector machines and its applications in chemistry , 2009 .

[89]  Sebastian G. Rohrer,et al.  Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based on PubChem Bioactivity Data , 2009, J. Chem. Inf. Model..

[90]  Zhide Hu,et al.  Prediction of fungicidal activities of rice blast disease based on least-squares support vector machines and project pursuit regression. , 2008, Journal of agricultural and food chemistry.

[91]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[92]  O. Dym,et al.  Thermal stabilization of the protozoan Entamoeba histolytica alcohol dehydrogenase by a single proline substitution , 2008, Proteins.

[93]  Koji Yasuda,et al.  Accelerating Density Functional Calculations with Graphics Processing Unit. , 2008, Journal of chemical theory and computation.

[94]  Weida Tong,et al.  Mold2, Molecular Descriptors from 2D Structures for Chemoinformatics and Toxicoinformatics , 2008, J. Chem. Inf. Model..

[95]  Yang Zhang Progress and challenges in protein structure prediction. , 2008, Current opinion in structural biology.

[96]  Oliver F. Lange,et al.  Consistent blind protein structure generation from NMR chemical shift data , 2008, Proceedings of the National Academy of Sciences.

[97]  Carsten Kutzner,et al.  GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation. , 2008, Journal of chemical theory and computation.

[98]  James C. Phillips,et al.  Accelerating molecular modeling applications with graphics processors , 2007, J. Comput. Chem..

[99]  Paola Gramatica,et al.  Principles of QSAR models validation: internal and external , 2007 .

[100]  Pierre Baldi,et al.  Improved residue contact prediction using support vector machines and a large feature set , 2007, BMC Bioinformatics.

[101]  A. Sali,et al.  Statistical potential for assessment and prediction of protein structures , 2006, Protein science : a publication of the Protein Society.

[102]  Shawn T. Brown,et al.  Advances in methods and algorithms in a modern quantum chemistry program package. , 2006, Physical chemistry chemical physics : PCCP.

[103]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[104]  David E. Kim,et al.  Physically realistic homology models built with ROSETTA can be more accurate than their templates. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[105]  M. Shara,et al.  Predicting physical properties of ionic liquids. , 2006, Physical chemistry chemical physics : PCCP.

[106]  Laxmikant V. Kalé,et al.  Scalable molecular dynamics with NAMD , 2005, J. Comput. Chem..

[107]  Holger Gohlke,et al.  The Amber biomolecular simulation programs , 2005, J. Comput. Chem..

[108]  Burkhard Rost,et al.  PROFcon: novel prediction of long-range contacts , 2005, Bioinform..

[109]  Guoli Wang,et al.  PISCES: recent improvements to a PDB sequence culling server , 2005, Nucleic Acids Res..

[110]  John Moult,et al.  A decade of CASP: progress, bottlenecks and prognosis in protein structure prediction. , 2005, Current opinion in structural biology.

[111]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[112]  Gunnar Rätsch,et al.  Classifying 'Drug-likeness' with Kernel-Based Learning Methods , 2005, J. Chem. Inf. Model..

[113]  J. Skolnick,et al.  Development and large scale benchmark testing of the PROSPECTOR_3 threading algorithm , 2004, Proteins.

[114]  P. Wolynes Energy landscapes and solved protein–folding problems , 2004, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[115]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[116]  Patrice Koehl,et al.  The ASTRAL Compendium in 2004 , 2003, Nucleic Acids Res..

[117]  Murray Campbell,et al.  Deep Blue , 2002, Artif. Intell..

[118]  Jens Meiler,et al.  Generation and evaluation of dimension-reduced amino acid parameter representations by artificial neural networks , 2001 .

[119]  B. Rost Review: protein secondary structure prediction continues to rise. , 2001, Journal of structural biology.

[120]  Frank R. Burden,et al.  Use of Automatic Relevance Determination in QSAR Studies Using Bayesian Neural Networks , 2000, J. Chem. Inf. Comput. Sci..

[121]  F. Burden,et al.  Robust QSAR models using Bayesian regularized neural networks. , 1999, Journal of medicinal chemistry.

[122]  Ajay,et al.  Can we learn to distinguish between "drug-like" and "nondrug-like" molecules? , 1998, Journal of medicinal chemistry.

[123]  Thomas L. Madden,et al.  Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. , 1997, Nucleic acids research.

[124]  M. Karelson,et al.  Quantum-Chemical Descriptors in QSAR/QSPR Studies. , 1996, Chemical reviews.

[125]  W. Guida,et al.  The art and practice of structure‐based drug design: A molecular modeling perspective , 1996, Medicinal research reviews.

[126]  Mark S. Gordon,et al.  General atomic and molecular electronic structure system , 1993, J. Comput. Chem..

[127]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[128]  Arthur E. Bryson,et al.  OPTIMAL PROGRAMMING PROBLEMS WITH INEQUALITY CONSTRAINTS , 1963 .

[129]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[130]  A. Tramontano,et al.  Evaluation of residue–residue contact predictions in CASP9 , 2011, Proteins.

[131]  B. Roux,et al.  Simulation of Osmotic Pressure in Concentrated Aqueous Salt Solutions , 2010 .

[132]  Kevin Karplus,et al.  Contact prediction using mutual information and neural nets , 2007, Proteins.

[133]  Yoshua Bengio,et al.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies , 2001 .

[134]  P Fariselli,et al.  Progress in predicting inter‐residue contacts of proteins with neural networks and correlated mutations , 2001, Proteins.

[135]  J. Onuchic,et al.  Theory of protein folding: the energy landscape perspective. , 1997, Annual review of physical chemistry.

[136]  M. Karelson,et al.  QSPR: the correlation and quantitative prediction of chemical and physical properties from structure , 1995 .

[137]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[138]  Marco Tulio Ribeiro,et al.  Association for Computational Linguistics " Why Should I Trust You? " Explaining the Predictions of Any Classifier , 2022 .