Assigning confidence to molecular property prediction

ABSTRACT Introduction: Computational modeling has rapidly advanced over the last decades. Recently, machine learning has emerged as a powerful and cost-effective strategy to learn from existing datasets and perform predictions on unseen molecules. Accordingly, the explosive rise of data-driven techniques raises an important question: What confidence can be assigned to molecular property predictions and what techniques can be used? Areas covered: The authors discuss popular strategies for predicting molecular properties, their corresponding uncertainty sources and methods to quantify uncertainty. First, the authors’ considerations for assessing confidence begin with dataset bias and size, data-driven property prediction and feature design. Next, the authors discuss property simulation via computations of binding affinity in detail. Lastly, they investigate how these uncertainties propagate to generative models, as they are usually coupled with property predictors. Expert opinion: Computational techniques are paramount to reduce the prohibitive cost of brute-force experimentation during exploration. The authors believe that assessing uncertainty in property prediction models is essential whenever closed-loop drug design campaigns relying on high-throughput virtual screening are deployed. Accordingly, considering sources of uncertainty leads to better-informed validations, more reliable predictions and more realistic expectations of the entire workflow. Overall, this increases confidence in the predictions and, ultimately, accelerates drug design.

[1]  Alán Aspuru-Guzik,et al.  Data-Driven Strategies for Accelerated Materials Design , 2021, Accounts of chemical research.

[2]  Sagi Eppel,et al.  Deep Molecular Dreaming: Inverse machine learning for de-novo molecular design and interpretability with surjective representations , 2020, Mach. Learn. Sci. Technol..

[3]  Gabriel dos Passos Gomes,et al.  Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery (STONED) algorithm for molecules using SELFIES , 2020, Chemical science.

[4]  Rafael Gómez-Bombarelli,et al.  Molecular machine learning with conformer ensembles , 2020, Mach. Learn. Sci. Technol..

[5]  B. L. de Groot,et al.  Non-equilibrium approach for binding free energies in cyclodextrins in SAMPL7: force fields and software , 2020, Journal of Computer-Aided Molecular Design.

[6]  Xavier Barril,et al.  Extended connectivity interaction features: improving binding affinity prediction through chemical description , 2020, Bioinform..

[7]  Piero Procacci,et al.  Methodological uncertainties in drug-receptor binding free energy predictions based on classical molecular dynamics. , 2020, Current opinion in structural biology.

[8]  D. Mobley,et al.  Challenges Encountered Applying Equilibrium and Nonequilibrium Binding Free Energy Calculations. , 2020, The journal of physical chemistry. B.

[9]  Marie L. Laury,et al.  AMOEBA binding free energies for the SAMPL7 TrimerTrip host–guest challenge , 2020, Journal of Computer-Aided Molecular Design.

[10]  Yurii S Moroz,et al.  ZINC20 - A Free Ultralarge-Scale Chemical Database for Ligand Discovery , 2020, J. Chem. Inf. Model..

[11]  Maho Nakata,et al.  PubChemQC PM6: Data Sets of 221 Million Molecules with Optimized Molecular Geometries and Electronic Properties , 2020, J. Chem. Inf. Model..

[12]  Ola Spjuth,et al.  Predicting with confidence: Using conformal prediction in drug discovery. , 2020, Journal of pharmaceutical sciences.

[13]  Brian Hie,et al.  Leveraging Uncertainty in Machine Learning Accelerates Biological Discovery and Design. , 2020, Cell systems.

[14]  Murat Cihan Sorkun,et al.  Pushing the limits of solubility prediction via quality-oriented data selection , 2020, iScience.

[15]  Henry B. Moss,et al.  Gaussian Process Molecule Property Prediction with FlowMO , 2020, ArXiv.

[16]  B. L. de Groot,et al.  On the importance of statistics in molecular simulations for thermodynamics, kinetics and simulation box size , 2020, eLife.

[17]  Niels Hansen,et al.  Combining Molecular Dynamics and Machine Learning to Predict Self-Solvation Free Energies and Limiting Activity Coefficients , 2020, J. Chem. Inf. Model..

[18]  R. Viola,et al.  Discovery of Novel Inhibitors of a Critical Brain Enzyme Using a Homology Model and a Deep Convolutional Neural Network. , 2020, Journal of medicinal chemistry.

[19]  Youjun Xu,et al.  Transfer Learning for Drug Discovery. , 2020, Journal of medicinal chemistry.

[20]  Woody Sherman,et al.  Rigorous Free Energy Simulations in Virtual Screening , 2020, J. Chem. Inf. Model..

[21]  Regina Barzilay,et al.  Uncertainty Quantification Using Neural Networks for Molecular Property Prediction , 2020, J. Chem. Inf. Model..

[22]  Artem Cherkasov,et al.  QSAR without borders. , 2020, Chemical Society reviews.

[23]  Avid M. Afzal,et al.  A Bayesian neural network for toxicity prediction , 2020, bioRxiv.

[24]  Barbara Pernici,et al.  Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction , 2020, J. Chem. Inf. Model..

[25]  David L. Mobley,et al.  Assessing the accuracy of octanol–water partition coefficient predictions in the SAMPL6 Part II log P Challenge , 2020, Journal of Computer-Aided Molecular Design.

[26]  Jincai Yang,et al.  Predicting or Pretending: Artificial Intelligence for Protein-Ligand Interactions Lack of Sufficiently Large and Unbiased Datasets , 2020, Frontiers in Pharmacology.

[27]  Andrew E. Brereton,et al.  Predicting drug properties with parameter-free machine learning: pareto-optimal embedded modeling (POEM) , 2020, Mach. Learn. Sci. Technol..

[28]  Alison J. Foster,et al.  Predicting Drug-Induced Liver Injury with Bayesian Machine Learning. , 2019, Chemical research in toxicology.

[29]  Eric J. Deeds,et al.  Machine learning classification can reduce false positives in structure-based virtual screening , 2020, Proceedings of the National Academy of Sciences.

[30]  Rommie E. Amaro,et al.  D3R grand challenge 4: blind prediction of protein–ligand poses, affinity rankings, and relative binding free energies , 2019, Journal of Computer-Aided Molecular Design.

[31]  B. L. de Groot,et al.  Large scale relative protein ligand binding affinities using non-equilibrium alchemy , 2019, Chemical science.

[32]  Pedro J. Ballester,et al.  Selecting machine-learning scoring functions for structure-based virtual screening. , 2019, Drug discovery today. Technologies.

[33]  Shae-Lynn J Lahey,et al.  Simulating protein–ligand binding with neural network potentials† , 2019, Chemical science.

[34]  G. Hunault,et al.  Dataset’s chemical diversity limits the generalizability of machine learning predictions , 2019, Journal of Cheminformatics.

[35]  Willem Waegeman,et al.  Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods , 2019, Machine Learning.

[36]  Denis Fourches,et al.  Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT , 2019, Journal of Cheminformatics.

[37]  Joseph A Morrone,et al.  Combining Docking Pose Rank and Structure with Deep Learning Improves Protein-Ligand Binding Mode Prediction over a Baseline Docking Approach , 2019, J. Chem. Inf. Model..

[38]  Alán Aspuru-Guzik,et al.  Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space , 2019, ICLR.

[39]  Andreas Bender,et al.  Concepts and Applications of Conformal Prediction in Computational Drug Discovery , 2019, ArXiv.

[40]  Murat Cihan Sorkun,et al.  AqSolDB, a curated reference set of aqueous solubility and 2D descriptors for a diverse set of compounds , 2019, Scientific Data.

[41]  Yaoqi Zhou,et al.  DLIGAND2: an improved knowledge-based energy function for protein–ligand interactions using the distance-scaled, finite, ideal-gas reference state , 2019, Journal of Cheminformatics.

[42]  Seongok Ryu,et al.  A Bayesian graph convolutional network for reliable prediction of molecular properties with uncertainty quantification , 2019, Chemical science.

[43]  Yao Zhang,et al.  Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. , 2019 .

[44]  O. Engkvist,et al.  A de novo molecular generation method using latent vector based generative adversarial network , 2019, J. Cheminformatics.

[45]  Pascal Friederich,et al.  Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation , 2019, Mach. Learn. Sci. Technol..

[46]  F. S. Costabal,et al.  Machine learning in drug development: Characterizing the effect of 30 drugs on the QT interval using Gaussian process regression, sensitivity analysis, and uncertainty quantification. , 2019, Computer methods in applied mechanics and engineering.

[47]  Teresa Head-Gordon,et al.  Systematic Optimization of Water Models Using Liquid/Vapor Surface Tension Data. , 2019, The journal of physical chemistry. B.

[48]  Jonas Boström,et al.  Deep Reinforcement Learning for Multiparameter Optimization in de novo Drug Design , 2019, J. Chem. Inf. Model..

[49]  Maho Nakata,et al.  PubChemQC PM6: Data Sets of 221 Million Molecules with Optimized Molecular Geometries and Electronic Properties , 2019, J. Chem. Inf. Model..

[50]  Viktor Hornak,et al.  Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structure-based virtual screening , 2019, PloS one.

[51]  Matthias Rarey,et al.  In Need of Bias Control: Evaluating Chemical Data for Machine Learning in Structure-Based Virtual Screening , 2019, J. Chem. Inf. Model..

[52]  Masakazu Sekijima,et al.  Improved Method of Structure-Based Virtual Screening via Interaction-Energy-Based Learning , 2019, J. Chem. Inf. Model..

[53]  Krzysztof Rataj,et al.  Mol-CycleGAN: a generative model for molecular optimization , 2019, Journal of Cheminformatics.

[54]  Alpha A. Lee,et al.  Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning , 2019, Chemical science.

[55]  Andrew R. Leach,et al.  ChEMBL: towards direct deposition of bioassay data , 2018, Nucleic Acids Res..

[56]  Daniel M. Zuckerman,et al.  Best Practices for Quantification of Uncertainty and Sampling Quality in Molecular Simulations , 2018 .

[57]  Jan H. Jensen,et al.  A graph-based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space , 2018, Chemical science.

[58]  Li Li,et al.  Optimization of Molecules via Deep Reinforcement Learning , 2018, Scientific Reports.

[59]  David L Mobley,et al.  Escaping Atom Types in Force Fields Using Direct Chemical Perception. , 2018, Journal of chemical theory and computation.

[60]  Charlotte M. Deane,et al.  Protein Family-Specific Models Using Deep Neural Networks and Transfer Learning Improve Virtual Screening and Highlight the Need for More Data , 2018, J. Chem. Inf. Model..

[61]  A. Poso,et al.  Binding Affinity via Docking: Fact and Fiction , 2018, Molecules.

[62]  Alán Aspuru-Guzik,et al.  Inverse molecular design using machine learning: Generative models for matter engineering , 2018, Science.

[63]  Michael R Shirts,et al.  Testing for physical validity in molecular simulations , 2018, PloS one.

[64]  Caitlin C. Bannan,et al.  SAMPL6 challenge results from $$pK_a$$pKa predictions based on a general Gaussian process model , 2018, Journal of Computer-Aided Molecular Design.

[65]  Koji Tsuda,et al.  Population-based de novo molecule generation, using grammatical evolution , 2018, 1804.02134.

[66]  Teresa Head-Gordon,et al.  New developments in force fields for biomolecular simulations. , 2018, Current opinion in structural biology.

[67]  Andreas Bender,et al.  Maximizing gain in high-throughput screening using conformal prediction , 2018, Journal of Cheminformatics.

[68]  Gianni De Fabritiis,et al.  KDEEP: Protein-Ligand Absolute Binding Affinity Prediction via 3D-Convolutional Neural Networks , 2018, J. Chem. Inf. Model..

[69]  Abhinav Vishnu,et al.  Using Rule-Based Labels for Weak Supervised Learning: A ChemNet for Transferable Chemical Property Prediction , 2017, KDD.

[70]  Olexandr Isayev,et al.  Deep reinforcement learning for de novo drug design , 2017, Science Advances.

[71]  Stanley E Lazic,et al.  Predicting drug safety and communicating risk: benefits of a Bayesian approach , 2017, bioRxiv.

[72]  Izhar Wallach,et al.  Most Ligand-Based Benchmarks Measure Overfitting Rather than Accuracy , 2017, J. Chem. Inf. Model..

[73]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[74]  Alán Aspuru-Guzik,et al.  Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models , 2017, ArXiv.

[75]  Maho Nakata,et al.  PubChemQC Project: A Large-Scale First-Principles Electronic Structure Database for Data-Driven Chemistry , 2017, J. Chem. Inf. Model..

[76]  Ranadip Pal,et al.  IntegratedMRF: random forest‐based framework for integrating prediction from different data types , 2017, Bioinform..

[77]  Pedro J. Ballester,et al.  Performance of machine-learning scoring functions in structure-based virtual screening , 2017, Scientific Reports.

[78]  Joseph Gomes,et al.  Building a More Predictive Protein Force Field: A Systematic and Reproducible Route to AMBER-FB15. , 2017, The journal of physical chemistry. B.

[79]  G. Schneider,et al.  Active learning for computational chemogenomics. , 2017, Future medicinal chemistry.

[80]  Joseph Gomes,et al.  MoleculeNet: a benchmark for molecular machine learning† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c7sc02664a , 2017, Chemical science.

[81]  Andreas Bender,et al.  Modelling compound cytotoxicity using conformal prediction and PubChem HTS data. , 2017, Toxicology research.

[82]  David Ryan Koes,et al.  Protein-Ligand Scoring with Convolutional Neural Networks , 2016, Journal of chemical information and modeling.

[83]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[84]  J S Smith,et al.  ANI-1: an extensible neural network potential with DFT accuracy at force field computational cost , 2016, Chemical science.

[85]  Alán Aspuru-Guzik,et al.  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules , 2016, ACS central science.

[86]  Ping Li,et al.  A review on Gaussian Process Latent Variable Models , 2016, CAAI Trans. Intell. Technol..

[87]  Hao Wu,et al.  Multiensemble Markov models of molecular thermodynamics and kinetics , 2016, Proceedings of the National Academy of Sciences.

[88]  L. Vosshall,et al.  Olfactory perception of chemically diverse molecules , 2016, bioRxiv.

[89]  Michael J. Bodkin,et al.  Accurate calculation of the absolute free energy of binding for drug molecules , 2015, Chemical science.

[90]  Marc C Nicklaus,et al.  Tautomerism of Warfarin: Combined Chemoinformatics, Quantum Chemical, and NMR Investigation. , 2015, The Journal of organic chemistry.

[91]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[92]  Dong-Sheng Cao,et al.  In silico evaluation of logD7.4 and comparison with other prediction methods , 2015 .

[93]  Gisbert Schneider,et al.  Active-learning strategies in computer-assisted drug discovery. , 2015, Drug discovery today.

[94]  Walid Gomaa,et al.  Machine learning in computational docking , 2015, Artif. Intell. Medicine.

[95]  Jennifer L. Knight,et al.  Accurate and reliable prediction of relative ligand binding potency in prospective drug discovery by way of a modern free-energy calculation protocol and force field. , 2015, Journal of the American Chemical Society.

[96]  Jie Li,et al.  PDB-wide collection of binding data: current status of the PDBbind database , 2015, Bioinform..

[97]  Gerhard Hummer,et al.  Free energies from dynamic weighted histogram analysis using unbiased Markov state model. , 2015, Journal of chemical theory and computation.

[98]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[99]  Pavlo O. Dral,et al.  Quantum chemistry structures and properties of 134 kilo molecules , 2014, Scientific Data.

[100]  Maykel Cruz-Monteagudo,et al.  Activity cliffs in drug discovery: Dr Jekyll or Mr Hyde? , 2014, Drug discovery today.

[101]  David L. Mobley,et al.  FreeSolv: a database of experimental and calculated hydration free energies, with input files , 2014, Journal of Computer-Aided Molecular Design.

[102]  Zhihai Liu,et al.  Comparative Assessment of Scoring Functions on an Updated Benchmark: 2. Evaluation Methods and General Results , 2014, J. Chem. Inf. Model..

[103]  Scott Boyer,et al.  Introducing Conformal Prediction in Predictive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination , 2014, J. Chem. Inf. Model..

[104]  Vijay S Pande,et al.  Building Force Fields: An Automatic, Systematic, and Reproducible Approach. , 2014, The journal of physical chemistry letters.

[105]  Eric T. Kim,et al.  Correction to How Does a Drug Molecule Find its Target Binding Site , 2014 .

[106]  Scott Boyer,et al.  Assessment of Machine Learning Reliability Methods for Quantifying the Applicability Domain of QSAR Regression Models , 2014, J. Chem. Inf. Model..

[107]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[108]  Wolfgang Marquardt,et al.  What is Wrong with Quantitative Structure-Property Relations Models Based on Three-Dimensional Descriptors? , 2012, J. Chem. Inf. Model..

[109]  Burr Settles,et al.  Active Learning , 2012, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[110]  Michael M. Mysinger,et al.  Directory of Useful Decoys, Enhanced (DUD-E): Better Ligands and Decoys for Better Benchmarking , 2012, Journal of medicinal chemistry.

[111]  Luis Pinheiro,et al.  A Bayesian Approach to in Silico Blood-Brain Barrier Penetration Modeling , 2012, J. Chem. Inf. Model..

[112]  Ajay N. Jain,et al.  Surflex-Dock: Docking benchmarks and real-world application , 2012, Journal of Computer-Aided Molecular Design.

[113]  Rhiju Das,et al.  Are Protein Force Fields Getting Better? A Systematic Benchmark on 524 Diverse NMR Measurements. , 2012, Journal of chemical theory and computation.

[114]  Robert P. Sheridan,et al.  Three Useful Dimensions for Domain Applicability in QSAR Models Using Random Forest , 2012, J. Chem. Inf. Model..

[115]  Evan Bolton,et al.  PubChem's BioAssay Database , 2011, Nucleic Acids Res..

[116]  Eric T. Kim,et al.  How does a drug molecule find its target binding site? , 2011, Journal of the American Chemical Society.

[117]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[118]  Pierre Tufféry,et al.  Frog2: Efficient 3D conformation ensemble generator for small compounds , 2010, Nucleic Acids Res..

[119]  Neil D. Lawrence,et al.  Bayesian Gaussian Process Latent Variable Model , 2010, AISTATS.

[120]  Margaret E. Johnson,et al.  Current status of the AMOEBA polarizable force field. , 2010, The journal of physical chemistry. B.

[121]  Arthur J. Olson,et al.  AutoDock Vina: Improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading , 2009, J. Comput. Chem..

[122]  Zhihai Liu,et al.  Comparative Assessment of Scoring Functions on a Diverse Test Set , 2009, J. Chem. Inf. Model..

[123]  Michael R. Shirts,et al.  Statistically optimal analysis of samples from multiple equilibrium states. , 2008, The Journal of chemical physics.

[124]  Scott D. Kahn,et al.  Current Status of Methods for Defining the Applicability Domain of (Quantitative) Structure-Activity Relationships , 2005, Alternatives to laboratory animals : ATLA.

[125]  Renxiao Wang,et al.  The PDBbind database: collection of binding affinities for protein-ligand complexes with known three-dimensional structures. , 2004, Journal of medicinal chemistry.

[126]  John S. Delaney,et al.  ESOL: Estimating Aqueous Solubility Directly from Molecular Structure , 2004, J. Chem. Inf. Model..

[127]  Matthew P. Repasky,et al.  Glide: a new approach for rapid, accurate docking and scoring. 1. Method and assessment of docking accuracy. , 2004, Journal of medicinal chemistry.

[128]  Hege S. Beard,et al.  Glide: a new approach for rapid, accurate docking and scoring. 2. Enrichment factors in database screening. , 2004, Journal of medicinal chemistry.

[129]  Richard D. Taylor,et al.  Improved protein–ligand docking using GOLD , 2003, Proteins.

[130]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[131]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[132]  G. Crooks Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences. , 1999, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[133]  R. Swendsen,et al.  THE weighted histogram analysis method for free‐energy calculations on biomolecules. I. The method , 1992 .

[134]  David Weininger,et al.  SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules , 1988, J. Chem. Inf. Comput. Sci..

[135]  Charles H. Bennett,et al.  Efficient estimation of free energy differences from Monte Carlo data , 1976 .

[136]  R. Zwanzig High‐Temperature Equation of State by a Perturbation Method. I. Nonpolar Gases , 1954 .

[137]  J. Kirkwood Statistical Mechanics of Fluid Mixtures , 1935 .

[138]  AkshatKumar Nigam,et al.  Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning , 2020, ArXiv.

[139]  Takeshi Okadome,et al.  Bayesian Kernel Regression for Noisy Inputs Based on Nadaraya-Watson Estimator Constructed from Noiseless Training Data , 2020, Adv. Data Sci. Adapt. Anal..

[140]  Alan Grossfield,et al.  Best Practices for Quantification of Uncertainty and Sampling Quality in Molecular Simulations [Article v1.0]. , 2018, Living journal of computational molecular science.

[141]  Alán Aspuru-Guzik,et al.  Optimizing distributions over molecular space. An Objective-Reinforced Generative Adversarial Network for Inverse-design Chemistry (ORGANIC) , 2017 .

[142]  Shuai Liu,et al.  D3R Grand Challenge 2: blind prediction of protein–ligand poses, affinity rankings, and relative binding free energies , 2017, Journal of Computer-Aided Molecular Design.

[143]  N. Xiao Lipophilicity Dataset - logD7.4 of 1,130 Compounds , 2017 .

[144]  S. Rigatti Random Forest. , 2017, Journal of insurance medicine.

[145]  Jian Yin,et al.  Lessons learned from comparing molecular dynamics engines on the SAMPL5 dataset , 2016, bioRxiv.

[146]  David L. Mobley,et al.  Alchemical Free Energy Calculations : Ready for Prime Time ? , 2016 .

[147]  Vytautas Gapsys,et al.  Calculation of binding free energies. , 2015, Methods in molecular biology.

[148]  D. Mobley,et al.  A simple method for automated equilibration detection in molecular simulations , 2015 .

[149]  David L Mobley,et al.  An introduction to best practices in free energy calculations. , 2013, Methods in molecular biology.

[150]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[151]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[152]  David L. Mobley,et al.  Chapter 4 Alchemical Free Energy Calculations: Ready for Prime Time? , 2007 .

[153]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[154]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[155]  D. Heiss-Czedik An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[156]  A. W.,et al.  Journal of chemical information and computer sciences. , 1995, Environmental science & technology.

[157]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).