Performance of Deep and Shallow Neural Networks, the Universal Approximation Theorem, Activity Cliffs, and QSAR

Neural networks have generated valuable Quantitative Structure‐Activity/Property Relationships (QSAR/QSPR) models for a wide variety of small molecules and materials properties. They have grown in sophistication and many of their initial problems have been overcome by modern mathematical techniques. QSAR studies have almost always used so‐called “shallow” neural networks in which there is a single hidden layer between the input and output layers. Recently, a new and potentially paradigm‐shifting type of neural network based on Deep Learning has appeared. Deep learning methods have generated impressive improvements in image and voice recognition, and are now being applied to QSAR and QSAR modelling. This paper describes the differences in approach between deep and shallow neural networks, compares their abilities to predict the properties of test sets for 15 large drug data sets (the kaggle set), discusses the results in terms of the Universal Approximation theorem for neural networks, and describes how DNN may ameliorate or remove troublesome “activity cliffs” in QSAR data sets.

[1]  Yi Li,et al.  Gene expression inference with deep learning , 2015, bioRxiv.

[2]  Frank R Burden,et al.  Quantitative structure-property relationship modeling of diverse materials properties. , 2012, Chemical reviews.

[3]  F. Burden,et al.  Robust QSAR models using Bayesian regularized neural networks. , 1999, Journal of medicinal chemistry.

[4]  Mati Karelson,et al.  Have artificial neural networks met expectations in drug discovery as implemented in QSAR framework? , 2016, Expert opinion on drug discovery.

[5]  David A. Winkler,et al.  Beware of R2: Simple, Unambiguous Assessment of the Prediction Accuracy of QSAR and QSPR Models , 2015, J. Chem. Inf. Model..

[6]  Gisbert Schneider,et al.  Deep Learning in Drug Discovery , 2016, Molecular informatics.

[7]  Igor V Tetko,et al.  A renaissance of neural networks in drug discovery , 2016, Expert opinion on drug discovery.

[8]  S. Kauffman,et al.  Search strategies for applied molecular evolution. , 1995, Journal of theoretical biology.

[9]  Frank R. Burden,et al.  Relevance Vector Machines: Sparse Classification Methods for QSAR , 2015, J. Chem. Inf. Model..

[10]  David A. Winkler,et al.  Understanding the Roles of the "Two QSARs" , 2016, J. Chem. Inf. Model..

[11]  Gerald M. Maggiora,et al.  On Outliers and Activity Cliffs-Why QSAR Often Disappoints , 2006, J. Chem. Inf. Model..

[12]  David A Winkler,et al.  Neural networks as robust tools in drug lead discovery and development , 2004, Molecular biotechnology.

[13]  H. J. Mclaughlin,et al.  Learn , 2002 .

[14]  Andrey M. Kazennov,et al.  Impact of distance-based metric learning on classification and visualization model performance and structure–activity landscapes , 2014, Journal of Computer-Aided Molecular Design.

[15]  Bo-Han Su,et al.  Dependence of QSAR Models on the Selection of Trial Descriptor Sets: A Demonstration Using Nanotoxicity Endpoints of Decorated Nanotubes , 2013, J. Chem. Inf. Model..

[16]  D. Winkler,et al.  Discovery and Optimization of Materials Using Evolutionary Approaches. , 2016, Chemical reviews.

[17]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[18]  Frank R. Burden,et al.  New QSAR Methods Applied to Structure-Activity Mapping and Combinatorial Chemistry , 1999, J. Chem. Inf. Comput. Sci..

[19]  Robert P. Sheridan,et al.  Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships , 2015, J. Chem. Inf. Model..

[20]  Dave Winkler.  The broader applications of neural and genetic modelling methods. , 2001, Drug discovery today.

[21]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[22]  Frank R. Burden,et al.  Optimal Sparse Descriptor Selection for QSAR Using Bayesian Methods , 2009 .

[23]  David A. Winkler,et al.  Neural networks in ADME and toxicity prediction , 2004 .

[24]  Dave Winkler,et al.  Bayesian Regularization of Neural Networks , 2009, Artificial Neural Networks.

[25]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[26]  Maykel Cruz-Monteagudo,et al.  Activity cliffs in drug discovery: Dr Jekyll or Mr Hyde? , 2014, Drug discovery today.

[27]  Takéhiko Nakama,et al.  Comparisons of Single- and Multiple-Hidden-Layer Neural Networks , 2011, ISNN.

[28]  Ragu Bharadwaj,et al.  Crowdsourcing in pharma: a strategic framework. , 2015, Drug discovery today.

[29]  Frank R. Burden,et al.  An Optimal Self‐Pruning Neural Network and Nonlinear Descriptor Selection in QSAR , 2009 .

[30]  Stephen J. Capuzzi,et al.  QSAR Modeling of Tox21 Challenge Stress Response and Nuclear Receptor Signaling Toxicity Assays , 2016, Front. Environ. Sci..

[31]  T. Le,et al.  A Bright Future for Evolutionary Methods in Drug Design , 2015, ChemMedChem.

[32]  O. Stegle,et al.  Deep learning for computational biology , 2016, Molecular systems biology.

[33]  T Scior,et al.  How to recognize and workaround pitfalls in QSAR studies: a critical review. , 2009, Current medicinal chemistry.

[34]  Jürgen Bajorath,et al.  Exploring activity cliffs in medicinal chemistry. , 2012, Journal of medicinal chemistry.

[35]  Bin Chen,et al.  Comparison of Random Forest and Pipeline Pilot Naïve Bayes in Prospective QSAR Predictions , 2012, J. Chem. Inf. Model..

[36]  Jürgen Bajorath,et al.  Frequency of Occurrence and Potency Range Distribution of Activity Cliffs in Bioactive Compounds , 2012, J. Chem. Inf. Model..

[37]  Sean Ekins.  The Next Era: Deep Learning in Pharmaceutical Research , 2016, Pharmaceutical Research.

[38]  Stefano Moro,et al.  Pharmaceutical Perspectives of Nonlinear QSAR Strategies , 2010, J. Chem. Inf. Model..

[39]  Vera Kurková,et al.  Kolmogorov's theorem and multilayer neural networks , 1992, Neural Networks.

[40]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.