LiveBench‐2: Large‐scale automated evaluation of protein structure prediction servers

The aim of LiveBench is to provide a continuous evaluation of structure prediction servers to inform developers and users about the current state‐of‐the‐art structure prediction tools. LiveBench differs from other evaluation experiments because it is a large‐scale and a fully automated procedure. Since LiveBench‐1, which finished in April 2000, and related but independent CASP3 and CAFASP1 experiments, significant progress in the field has occurred. Some of the new developments have already been assessed at the recent CASP4 and CAFASP2 experiments (both independently of LiveBench), but others have not been observed yet because they entail developments carried out only recently. These include the availability of new servers (Pcons, FUGUE, and Coblath) and the enhancement of previously existing tools (mGenThreader, Sam‐T, and 3D‐PSSM), which illustrate the fast rate at which the field is advancing. Consequently, to keep in pace with the development, we present the results of the second large‐scale evaluation of protein structure prediction servers. Of the 11 fold recognition servers evaluated, two servers appear to be most sensitive. One of these is 3D‐PSSM, a server significantly improved after LiveBench‐1. The other top performer is the new consensus server Pcons, which significantly outperformed other servers in the specificity of predictions. LiveBench‐2 shows that the top performing servers are able to accurately recognize a fold for about one third of the “difficult” targets, a clear improvement over LiveBench‐1 results. Given that automated structure prediction is increasingly becoming a biologists companion, the guidelines drawn from the LiveBench experiments are likely to provide users with valuable and timely information for their prediction needs. Proteins 2001;Suppl 5:184–191. © 2002 Wiley‐Liss, Inc.

[1]  Marcin von Grotthuss,et al.  ORFeus: detection of distant homology using sequence profiles and predicted secondary structure , 2003, Nucleic Acids Res..

[2]  M. Sternberg,et al.  Enhanced genome annotation using structural profiles in the program 3D-PSSM. , 2000, Journal of molecular biology.

[3]  T. Hubbard,et al.  Critical assessment of methods of protein structure prediction (CASP)‐round V , 2003, Proteins.

[4]  Ying Xu,et al.  Protein Threading by Linear Programming , 2003, Pacific Symposium on Biocomputing.

[5]  A. Godzik,et al.  Comparison of sequence profiles. Strategies for structural predictions using sequence information , 2008, Protein science : a publication of the Protein Society.

[6]  T. N. Bhat,et al.  The Protein Data Bank , 2000, Nucleic Acids Res..

[7]  D Fischer,et al.  CAFASP‐1: Critical assessment of fully automated structure prediction methods , 1999, Proteins.

[8]  Daniel Fischer,et al.  3D‐SHOTGUN: A novel, cooperative, fold‐recognition meta‐predictor , 2003, Proteins.

[9]  W F van Gunsteren,et al.  Combined procedure of distance geometry and restrained molecular dynamics techniques for protein structure determination from nuclear magnetic resonance data: Application to the DNA binding domain of lac repressor from Escherichia coli , 1988, Proteins.

[10]  S. Bryant,et al.  Critical assessment of methods of protein structure prediction (CASP): Round II , 1997, Proteins.

[11]  Michael J. E. Sternberg,et al.  SAWTED: Structure Assignment With Text Description-Enhanced detection of remote homologues with automated SWISS-PROT annotation comparisons , 2000, Bioinform..

[12]  D Fischer,et al.  Hybrid fold recognition: combining sequence derived properties with evolutionary information. , 1999, Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing.

[13]  J A Swets,et al.  Better decisions through science. , 2000, Scientific American.

[14]  Roland L. Dunbrack,et al.  CAFASP2: The second critical assessment of fully automated structure prediction methods , 2001, Proteins.

[15]  Thomas L. Madden,et al.  Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. , 1997, Nucleic acids research.

[16]  C Kooperberg,et al.  Assembly of protein tertiary structures from fragments with similar local sequences using simulated annealing and Bayesian scoring functions. , 1997, Journal of molecular biology.

[17]  Tim J. P. Hubbard,et al.  SCOP: a Structural Classification of Proteins database , 1999, Nucleic Acids Res..

[18]  Marc A. Martí-Renom,et al.  EVA: continuous automatic evaluation of protein structure prediction servers , 2001, Bioinform..

[19]  Y Shan,et al.  Fold recognition and accurate query‐template alignment by a combination of PSI‐BLAST and threading , 2001, Proteins.

[20]  J Lundström,et al.  Pcons: A neural‐network–based consensus predictor that improves fold recognition , 2001, Protein science : a publication of the Protein Society.

[21]  Arne Elofsson,et al.  A study of quality measures for protein threading models , 2001, BMC Bioinformatics.

[22]  K Karplus,et al.  Predicting protein structure using only sequence information , 1999, Proteins.

[23]  D Fischer,et al.  LiveBench‐1: Continuous benchmarking of protein structure prediction servers , 2001, Protein science : a publication of the Protein Society.

[24]  Arne Elofsson,et al.  MaxSub: an automated measure for the assessment of protein structure prediction quality , 2000, Bioinform..

[25]  K Karplus,et al.  What is the value added by human intervention in protein structure prediction? , 2001, Proteins.

[26]  A. Elofsson,et al.  Can correct protein models be identified? , 2003, Protein science : a publication of the Protein Society.

[27]  A. Godzik,et al.  Regularities in interaction patterns of globular proteins. , 1993, Protein engineering.

[28]  David C. Jones,et al.  GenTHREADER: an efficient and reliable protein fold recognition method for genomic sequences. , 1999, Journal of molecular biology.

[29]  G J Williams,et al.  The Protein Data Bank: a computer-based archival file for macromolecular structures. , 1977, Journal of molecular biology.