Automated benchmarking of peptide-MHC class I binding predictions

MOTIVATION Numerous in silico methods predicting peptide binding to major histocompatibility complex (MHC) class I molecules have been developed over the last decades. However, the multitude of available prediction tools makes it non-trivial for the end-user to select which tool to use for a given task. To provide a solid basis on which to compare different prediction tools, we here describe a framework for the automated benchmarking of peptide-MHC class I binding prediction tools. The framework runs weekly benchmarks on data that are newly entered into the Immune Epitope Database (IEDB), giving the public access to frequent, up-to-date performance evaluations of all participating tools. To overcome potential selection bias in the data included in the IEDB, a strategy was implemented that suggests a set of peptides for which different prediction methods give divergent predictions as to their binding capability. Upon experimental binding validation, these peptides entered the benchmark study. RESULTS The benchmark has run for 15 weeks and includes evaluation of 44 datasets covering 17 MHC alleles and more than 4000 peptide-MHC binding measurements. Inspection of the results allows the end-user to make educated selections between participating tools. Of the four participating servers, NetMHCpan performed the best, followed by ANN, SMM and finally ARB. AVAILABILITY AND IMPLEMENTATION Up-to-date performance evaluations of each server can be found online at http://tools.iedb.org/auto_bench/mhci/weekly. All prediction tool developers are invited to participate in the benchmark. Sign-up instructions are available at http://tools.iedb.org/auto_bench/mhci/join. CONTACT mniel@cbs.dtu.dk or bpeters@liai.org SUPPLEMENTARY INFORMATION Supplementary data are available at Bioinformatics online.

[1]  Hiroshi Mamitsuka,et al.  Toward more accurate pan-specific MHC-peptide binding prediction: a review of current methods and tools , 2011, Briefings Bioinform..

[2]  Clemencia Pinilla,et al.  Derivation of an amino acid similarity matrix for peptide:MHC binding and its application as a Bayesian prior , 2009, BMC Bioinformatics.

[3]  O. Lund,et al.  NetMHCpan, a method for MHC class I binding prediction beyond humans , 2008, Immunogenetics.

[4]  P. Kloetzel,et al.  The role of the proteasome in the generation of MHC class I ligands and immune responses , 2011, Cellular and Molecular Life Sciences.

[5]  Hau-San Wong,et al.  MHC binding prediction with KernelRLSpan and its variations. , 2014, Journal of immunological methods.

[6]  Morten Nielsen,et al.  A Community Resource Benchmarking Predictions of Peptide Binding to MHC-I Molecules , 2006, PLoS Comput. Biol..

[7]  Magdalini Moutaftsi,et al.  A consensus epitope prediction approach identifies the breadth of murine TCD8+-cell responses to vaccinia virus , 2006, Nature Biotechnology.

[8]  Morten Nielsen,et al.  Pan-specific MHC class I predictors: a benchmark of HLA class I pan-specific prediction methods , 2009, Bioinform..

[10]  Channa K. Hattotuwagama,et al.  Quantitative online prediction of peptide binding to the major histocompatibility complex. , 2004, Journal of molecular graphics & modelling.

[11]  Alessandro Sette,et al.  Generating quantitative models describing the sequence specificity of biological processes with the stabilized matrix method , 2005, BMC Bioinformatics.

[12]  O. Lund,et al.  NetMHCpan, a Method for Quantitative Predictions of Peptide Binding to Any HLA-A and -B Locus Protein of Known Sequence , 2007, PloS one.

[13]  Bjoern Peters,et al.  Automated generation and evaluation of specific MHC binding predictive tools: ARB matrix applications , 2005, Immunogenetics.

[14]  Philip E. Bourne,et al.  Immune epitope database analysis resource , 2012, Nucleic Acids Res..

[15]  Burkhard Rost,et al.  Evaluation of template‐based models in CASP8 with standard measures , 2009, Proteins.

[16]  A. Goldberg,et al.  Inhibitors of the proteasome block the degradation of most cell proteins and the generation of peptides presented on MHC class I molecules , 1994, Cell.

[17]  John Sidney,et al.  Measurement of MHC/Peptide Interactions by Gel Filtration , 1999, Current protocols in immunology.

[18]  Morten Nielsen,et al.  NetMHCcons: a consensus method for the major histocompatibility complex class I predictions , 2011, Immunogenetics.

[19]  V. Brusic,et al.  Evaluation of MHC class I peptide binding prediction servers: Applications for vaccine research , 2008, BMC Immunology.

[20]  Harris Papadopoulos,et al.  Machine learning competition in immunology - Prediction of HLA class I binding peptides. , 2011, Journal of immunological methods.

[21]  M. Jenkins,et al.  The Role of Naive T Cell Precursor Frequency and Recruitment in Dictating Immune Response Magnitude , 2012, The Journal of Immunology.

[22]  Marc A. Martí-Renom,et al.  EVA: continuous automatic evaluation of protein structure prediction servers , 2001, Bioinform..

[23]  Morten Nielsen,et al.  The PickPocket method for predicting binding specificities for receptors based on receptor pocket similarities: application to MHC-peptide binding , 2009, Bioinform..

[24]  J. Yewdell,et al.  Immunodominance in major histocompatibility complex class I-restricted T lymphocyte responses. , 1999, Annual review of immunology.

[25]  P. van Endert,et al.  Peptidases trimming MHC class I ligands. , 2013, Current opinion in immunology.

[26]  Alessandro Sette,et al.  The Immune Epitope Database 2.0 , 2009, Nucleic Acids Res..

[27]  Ji Wan,et al.  SVRMHC prediction server for MHC-binding peptides , 2006, BMC Bioinformatics.

[28]  P. Cresswell,et al.  Evidence that transporters associated with antigen processing translocate a major histocompatibility complex class I-binding peptide into the endoplasmic reticulum in an ATP-dependent manner. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[29]  Krzysztof Fidelis,et al.  CASP prediction center infrastructure and evaluation measures in CASP10 and CASP ROLL , 2014, Proteins.

[30]  Morten Nielsen,et al.  Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions , 2014, BMC Bioinformatics.

[31]  Vladimir Brusic,et al.  Evaluation of MHC-II peptide binding prediction servers: applications for vaccine research , 2008, BMC Bioinformatics.

[32]  N. Shastri,et al.  ERAAP customizes peptides for MHC class I molecules in the endoplasmic reticulum , 2002, Nature.

[33]  Morten Nielsen,et al.  NetMHC-3.0: accurate web accessible predictions of human, mouse and monkey MHC class I affinities for peptides of length 8–11 , 2008, Nucleic Acids Res..

[34]  Hongjun Bai,et al.  Assessment of template‐free modeling in CASP10 and ROLL , 2014, Proteins.

[35]  A. Goldberg,et al.  Degradation of cell proteins and the generation of MHC class I-presented peptides. , 1999, Annual review of immunology.

[36]  Christine Almunia,et al.  Selective identification of HLA-DP4 binding T cell epitopes encoded by the MAGE-A gene family , 2007, Cancer Immunology, Immunotherapy.