Assessing protein–ligand interaction scoring functions with the CASF-2013 benchmark

Scoring functions are a group of computational methods widely applied in structure-based drug design for fast evaluation of protein-ligand interactions. To date, a whole spectrum of scoring functions have been developed based on different assumptions or algorithms. Therefore, it is important to both the end users and the developers of scoring functions that their performance be objectively assessed. We have developed the comparative assessment of scoring functions (CASF) benchmark as an open-access solution for scoring function evaluation. The latest CASF-2013 benchmark enables evaluation of the so-called 'scoring power', 'ranking power', 'docking power', and 'screening power' of a given scoring function with a high-quality test set of 195 complexes formed between diverse protein molecules and their small-molecule ligands. Evaluation results of the standard scoring functions implemented in several mainstream software programs (including Schrödinger, MOE, Discovery Studio, SYBYL, and GOLD) are provided as reference. This benchmark has become popular among the scoring function community since its first release. In this protocol, we provide detailed descriptions of the data files included in the CASF-2013 package and step-by-step instructions on how to conduct the performance tests with the ready-to-use computer scripts included in the package. This protocol is expected to lower the technical hurdles in front of new and existing users of the CASF-2013 benchmark. On a standard desktop workstation, it takes roughly half an hour to complete the whole evaluation procedure for one scoring function, once the required inputs, i.e., the results computed on the test set, are ready to use.

[1]  Richard D. Smith,et al.  CSAR Data Set Release 2012: Ligands, Affinities, Complexes, and Docking Decoys , 2013, J. Chem. Inf. Model..

[2]  Maria Kontoyianni,et al.  Evaluation of docking performance: comparative data on docking algorithms. , 2004, Journal of medicinal chemistry.

[3]  T. Klabunde,et al.  Structure-based drug discovery using GPCR homology modeling: successful virtual screening for antagonists of the alpha1A adrenergic receptor. , 2005, Journal of medicinal chemistry.

[4]  Zhihai Liu,et al.  Comparative Assessment of Scoring Functions on a Diverse Test Set , 2009, J. Chem. Inf. Model..

[5]  M Rarey,et al.  Detailed analysis of scoring functions for virtual screening. , 2001, Journal of medicinal chemistry.

[6]  Richard D. Smith,et al.  CSAR Benchmark Exercise of 2010: Combined Evaluation Across All Submitted Scoring Functions , 2011, J. Chem. Inf. Model..

[7]  Jie Li,et al.  PDB-wide collection of binding data: current status of the PDBbind database , 2015, Bioinform..

[8]  Didier Rognan,et al.  Comparative evaluation of eight docking tools for docking and virtual screening accuracy , 2004, Proteins.

[9]  Michael G. Lerner,et al.  Binding MOAD (Mother Of All Databases) , 2005, Proteins.

[10]  Jin Li,et al.  On Evaluating Molecular-Docking Methods for Pose Prediction and Enrichment Factors , 2006, J. Chem. Inf. Model..

[11]  Luhua Lai,et al.  Further development and validation of empirical scoring functions for structure-based binding affinity prediction , 2002, J. Comput. Aided Mol. Des..

[12]  Jie Liu,et al.  Classification of Current Scoring Functions , 2015, J. Chem. Inf. Model..

[13]  David S. Goodsell,et al.  Automated docking using a Lamarckian genetic algorithm and an empirical binding free energy function , 1998 .

[14]  Todd J. A. Ewing,et al.  DOCK 4.0: Search strategies for automated molecular docking of flexible molecule databases , 2001, J. Comput. Aided Mol. Des..

[15]  E. Jaeger,et al.  Comparison of automated docking programs as virtual screening tools. , 2005, Journal of Medicinal Chemistry.

[16]  Renxiao Wang,et al.  The PDBbind database: methodologies and updates. , 2005, Journal of medicinal chemistry.

[17]  Richard D. Smith,et al.  CSAR Benchmark Exercise 2013: Evaluation of Results from a Combined Computational Protein Design, Docking, and Scoring/Ranking Challenge , 2016, J. Chem. Inf. Model..

[18]  Nikolay P. Todorov,et al.  The Influence of Variations of Ligand Protonation and Tautomerism on Protein-Ligand Recognition and Binding Energy Landscape , 2006, J. Chem. Inf. Model..

[19]  W Patrick Walters,et al.  A detailed comparison of current docking and scoring methods on systems of pharmaceutical relevance , 2004, Proteins.

[20]  Andrew C Good,et al.  Ranking poses in structure-based lead discovery and optimization: current trends in scoring function development. , 2007, Current opinion in drug discovery & development.

[21]  Shuichi Hirono,et al.  Comparison of Consensus Scoring Strategies for Evaluating Computational Models of Protein-Ligand Complexes , 2006, J. Chem. Inf. Model..

[22]  C. E. Peishoff,et al.  A critical assessment of docking programs and scoring functions. , 2006, Journal of medicinal chemistry.

[23]  Dan Li,et al.  Comprehensive evaluation of ten docking programs on a diverse set of protein-ligand complexes: the prediction accuracy of sampling power and scoring power. , 2016, Physical chemistry chemical physics : PCCP.

[24]  Ingo Muegge,et al.  Evaluation of docking/scoring approaches: A comparative study based on MMP3 inhibitors , 2000, J. Comput. Aided Mol. Des..

[25]  J. Bajorath,et al.  Docking and scoring in virtual screening for drug discovery: methods and applications , 2004, Nature Reviews Drug Discovery.

[26]  Renxiao Wang,et al.  The PDBbind database: collection of binding affinities for protein-ligand complexes with known three-dimensional structures. , 2004, Journal of medicinal chemistry.

[27]  Robin Taylor,et al.  Comparing protein–ligand docking programs is difficult , 2005, Proteins.

[28]  Robert P. Sheridan,et al.  Comparison of Topological, Shape, and Docking Methods in Virtual Screening , 2007, J. Chem. Inf. Model..

[29]  Jie Li,et al.  Comparative Assessment of Scoring Functions on an Updated Benchmark: 1. Compilation of the Test Set , 2014, J. Chem. Inf. Model..

[30]  Zhihai Liu,et al.  Evaluation of the performance of four molecular docking programs on a diverse set of protein‐ligand complexes , 2010, J. Comput. Chem..

[31]  Matthew P. Repasky,et al.  Glide: a new approach for rapid, accurate docking and scoring. 1. Method and assessment of docking accuracy. , 2004, Journal of medicinal chemistry.

[32]  Li Xing,et al.  Evaluation and application of multiple scoring functions for a virtual screening experiment , 2004, J. Comput. Aided Mol. Des..

[33]  A. Ortiz,et al.  Evaluation of docking functions for protein-ligand docking. , 2001, Journal of medicinal chemistry.

[34]  Yongbo Hu,et al.  Comparison of Several Molecular Docking Programs: Pose Prediction and Virtual Screening Accuracy , 2009, J. Chem. Inf. Model..

[35]  D. Rognan,et al.  Protein-based virtual screening of chemical databases. 1. Evaluation of different docking/scoring combinations. , 2000, Journal of medicinal chemistry.

[36]  Martin Stahl,et al.  The Use of Scoring Functions in Drug Discovery Applications , 2003 .

[37]  J M Blaney,et al.  A geometric approach to macromolecule-ligand interactions. , 1982, Journal of molecular biology.

[38]  Richard D. Smith,et al.  CSAR Benchmark Exercise 2011–2012: Evaluation of Results from Docking and Relative Ranking of Blinded Congeneric Series , 2013, J. Chem. Inf. Model..

[39]  Maria Kontoyianni,et al.  Evaluation of library ranking efficacy in virtual screening , 2005, J. Comput. Chem..

[40]  Tiziano Tuccinardi,et al.  Extensive Consensus Docking Evaluation for Ligand Pose Prediction and Virtual Screening Studies , 2014, J. Chem. Inf. Model..

[41]  Gisbert Schneider,et al.  Computer-based de novo design of drug-like molecules , 2005, Nature Reviews Drug Discovery.

[42]  Martin Korth,et al.  Comparison of molecular mechanics, semi-empirical quantum mechanical, and density functional theory methods for scoring protein-ligand interactions. , 2013, The journal of physical chemistry. B.

[43]  David S. Goodsell,et al.  AutoDock4 and AutoDockTools4: Automated docking with selective receptor flexibility , 2009, J. Comput. Chem..

[44]  S. Balaz,et al.  A practical approach to docking of zinc metalloproteinase inhibitors. , 2004, Journal of molecular graphics & modelling.

[45]  Douglas R. Houston,et al.  Consensus Docking: Improving the Reliability of Docking in a Virtual Screening Context , 2013, J. Chem. Inf. Model..

[46]  John B. O. Mitchell,et al.  Predicting protein-ligand binding affinities: a low scoring game? , 2004, Organic & biomolecular chemistry.

[47]  Hege S. Beard,et al.  Glide: a new approach for rapid, accurate docking and scoring. 2. Enrichment factors in database screening. , 2004, Journal of medicinal chemistry.

[48]  Kenji Onodera,et al.  Evaluations of Molecular Docking Programs for Virtual Screening , 2007, J. Chem. Inf. Model..

[49]  M. Murcko,et al.  Consensus scoring: A method for obtaining improved hit rates from docking databases of three-dimensional structures into proteins. , 1999, Journal of medicinal chemistry.

[50]  Richard D. Smith,et al.  Binding MOAD, a high-quality protein–ligand database , 2007, Nucleic Acids Res..

[51]  Dariusz Plewczynski,et al.  Can we trust docking results? Evaluation of seven commonly used programs on PDBbind database , 2011, J. Comput. Chem..

[52]  D. J. Price,et al.  Assessing scoring functions for protein-ligand interactions. , 2004, Journal of medicinal chemistry.

[53]  Ajay N. Jain.  Surflex-Dock 2.1: Robust performance from ligand energetic modeling, ring flexibility, and knowledge-based search , 2007, J. Comput. Aided Mol. Des..

[54]  Richard D. Smith,et al.  CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma , 2016, J. Chem. Inf. Model..

[55]  Zhihai Liu,et al.  Comparative Assessment of Scoring Functions on an Updated Benchmark: 2. Evaluation Methods and General Results , 2014, J. Chem. Inf. Model..

[56]  Pedro J Ballester,et al.  Machine‐learning scoring functions to improve structure‐based binding affinity prediction and virtual screening , 2015, Wiley interdisciplinary reviews. Computational molecular science.

[57]  Renxiao Wang,et al.  Comparative evaluation of 11 scoring functions for molecular docking. , 2003, Journal of medicinal chemistry.

[58]  P Willett,et al.  Development and validation of a genetic algorithm for flexible docking. , 1997, Journal of molecular biology.

[59]  Richard D. Smith,et al.  CSAR Benchmark Exercise of 2010: Selection of the Protein–Ligand Complexes , 2011, J. Chem. Inf. Model..

[60]  Jeffrey Skolnick,et al.  Assessment of programs for ligand binding affinity prediction , 2008, J. Comput. Chem..

[61]  Thomas E. Exner,et al.  Influence of Protonation, Tautomeric, and Stereoisomeric States on Protein-Ligand Docking Results , 2009, J. Chem. Inf. Model..

[62]  Shaomeng Wang,et al.  An Extensive Test of 14 Scoring Functions Using the PDBbind Refined Set of 800 Protein-Ligand Complexes , 2004, J. Chem. Inf. Model..

[63]  Ajay N. Jain.  Bias, reporting, and sharing: computational evaluations of docking methods , 2008, J. Comput. Aided Mol. Des..

[64]  Natasja Brooijmans,et al.  Molecular recognition and docking algorithms. , 2003, Annual review of biophysics and biomolecular structure.

[65]  Brian K Shoichet,et al.  Prediction of protein-ligand interactions. Docking and scoring: successes and gaps. , 2006, Journal of medicinal chemistry.

[66]  Haruki Nakamura,et al.  Announcing the worldwide Protein Data Bank , 2003, Nature Structural Biology.

[67]  Martin Stahl,et al.  Scoring functions for protein-ligand interactions: a critical perspective. , 2004, Drug discovery today. Technologies.

[68]  F. Lombardo,et al.  Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings , 1997 .

[69]  Richard A. Friesner,et al.  Comparative Performance of Several Flexible Docking Programs and Scoring Functions: Enrichment Studies for a Diverse Set of Pharmaceutically Relevant Targets , 2007, J. Chem. Inf. Model..

[70]  Ruben Abagyan,et al.  Comparative study of several algorithms for flexible ligand docking , 2003, J. Comput. Aided Mol. Des..

[71]  Matthias Rarey,et al.  Small Molecule Docking and Scoring , 2001 .

[72]  Peter S Kutchukian,et al.  De novo design: balancing novelty and confined chemical space , 2010, Expert opinion on drug discovery.

[73]  Ajay N. Jain.  Surflex: fully automatic flexible molecular docking using a molecular similarity-based search engine. , 2003, Journal of medicinal chemistry.

[74]  Richard D. Smith,et al.  Recent improvements to Binding MOAD: a resource for protein–ligand binding affinities and structures , 2014, Nucleic Acids Res..

[75]  D. Fairlie,et al.  Comparing sixteen scoring functions for predicting biological activities of ligands for protein targets. , 2015, Journal of molecular graphics & modelling.