GeneNetWeaver: in silico benchmark generation and performance profiling of network inference methods

MOTIVATION Over the last decade, numerous methods have been developed for inference of regulatory networks from gene expression data. However, accurate and systematic evaluation of these methods is hampered by the difficulty of constructing adequate benchmarks and the lack of tools for a differentiated analysis of network predictions on such benchmarks. RESULTS Here, we describe a novel and comprehensive method for in silico benchmark generation and performance profiling of network inference methods available to the community as an open-source software called GeneNetWeaver (GNW). In addition to the generation of detailed dynamical models of gene regulatory networks to be used as benchmarks, GNW provides a network motif analysis that reveals systematic prediction errors, thereby indicating potential ways of improving inference methods. The accuracy of network inference methods is evaluated using standard metrics such as precision-recall and receiver operating characteristic curves. We show how GNW can be used to assess the performance and identify the strengths and weaknesses of six inference methods. Furthermore, we used GNW to provide the international Dialogue for Reverse Engineering Assessments and Methods (DREAM) competition with three network inference challenges (DREAM3, DREAM4 and DREAM5). AVAILABILITY GNW is available at http://gnw.sourceforge.net along with its Java source code, user manual and supporting data. SUPPLEMENTARY INFORMATION Supplementary data are available at Bioinformatics online. CONTACT dario.floreano@epfl.ch.

[1]  G. K. Ackers,et al.  Quantitative model for gene regulation by lambda phage repressor. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[2]  Ronald W. Davis,et al.  Quantitative Monitoring of Gene Expression Patterns with a Complementary DNA Microarray , 1995, Science.

[3]  D. Lockhart,et al.  Expression monitoring by hybridization to high-density oligonucleotide arrays , 1996, Nature Biotechnology.

[4]  D. Gillespie.  The chemical Langevin equation , 2000 .

[5]  L. Serrano,et al.  Engineering stability in gene networks by autoregulation , 2000, Nature.

[6]  L Nyström,et al.  Statistical Analysis , 2008, Encyclopedia of Social Network Analysis and Mining.

[7]  S. Shen-Orr,et al.  Network motifs in the transcriptional regulation network of Escherichia coli , 2002, Nature Genetics.

[8]  Hierarchical Organization of Modularity in Metabolic Networks Supporting Online Material , 2002 .

[9]  S. Shen-Orr,et al.  Network motifs: simple building blocks of complex networks. , 2002, Science.

[10]  A. Barabasi,et al.  Hierarchical Organization of Modularity in Metabolic Networks , 2002, Science.

[11]  Pedro Mendes,et al.  Artificial gene networks for objective comparison of analysis algorithms , 2003, ECCB.

[12]  Satoru Miyano,et al.  Inferring gene networks from time series microarray data using dynamic Bayesian networks , 2003, Briefings Bioinform..

[13]  F. Doyle,et al.  A benchmark for methods in reverse engineering and model discrimination: problem formulation and solutions. , 2004, Genome research.

[14]  Paul P. Wang,et al.  Advances to Bayesian network inference for generating causal networks from observational biological data , 2004, Bioinform..

[15]  J. Roach,et al.  Statistical analysis of MPSS measurements: application to the study of LPS-activated macrophage gene expression. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Gustavo Stolovitzky,et al.  Reconstructing biological networks using conditional correlation analysis , 2005, Bioinform..

[17]  Kathleen Marchal,et al.  SynTReN: a generator of synthetic gene expression data for design and analysis of structure learning algorithms , 2006, BMC Bioinformatics.

[18]  Richard Bonneau,et al.  The Inferelator: an algorithm for learning parsimonious regulatory networks from systems-biology data sets de novo , 2006, Genome Biology.

[19]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[20]  E. O’Shea,et al.  Quantification of protein half-lives in the budding yeast proteome , 2006, Proceedings of the National Academy of Sciences.

[21]  Chris Wiggins,et al.  ARACNE: An Algorithm for the Reconstruction of Gene Regulatory Networks in a Mammalian Cellular Context , 2004, BMC Bioinformatics.

[22]  J. Collins,et al.  Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression Profiles , 2007, PLoS biology.

[23]  D. di Bernardo,et al.  How to infer gene networks from expression profiles , 2007, Molecular systems biology.

[24]  Margaret Werner-Washburne,et al.  A system for generating transcription regulatory networks with combinatorial control of transcription , 2008, Bioinform..

[25]  B. Williams,et al.  Mapping and quantifying mammalian transcriptomes by RNA-Seq , 2008, Nature Methods.

[26]  Claudio Cobelli,et al.  A Gene Network Simulator to Assess Reverse Engineering Algorithms , 2009, Annals of the New York Academy of Sciences.

[27]  D. Bernardo,et al.  A Yeast Synthetic Network for In Vivo Assessment of Reverse-Engineering and Modeling Approaches , 2009, Cell.

[28]  Yong Li,et al.  ReTRN: a retriever of real transcriptional regulatory network and expression data for evaluating structure learning algorithm. , 2009, Genomics.

[29]  Michael R. Brent,et al.  Benchmarking regulatory network reconstruction with GRENDEL , 2009, Bioinform..

[30]  Harri Lähdesmäki,et al.  Learning gene regulatory networks from gene expression measurements using non-parametric molecular kinetics , 2009, Bioinform..

[31]  Diogo M. Camacho,et al.  Systems Biology Strikes Gold , 2009, Cell.

[32]  Dario Floreano,et al.  Generating Realistic In Silico Gene Networks for Performance Assessment of Reverse Engineering Methods , 2009, J. Comput. Biol..

[33]  Ralf Herwig,et al.  GeNGe: systematic generation of gene regulatory networks , 2009, Bioinform..

[34]  Dario Floreano,et al.  Combining Multiple Results of a Reverse‐Engineering Algorithm: Application to the DREAM Five‐Gene Network Challenge , 2009, Annals of the New York Academy of Sciences.

[35]  N. D. Clarke,et al.  Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges , 2010, PloS one.

[36]  Chaoyang Zhang,et al.  Time lagged information theoretic approaches to the reverse engineering of gene regulatory networks , 2010, BMC Bioinformatics.

[37]  Robert J. Flassig,et al.  TRANSWESD: inferring cellular networks with transitive reduction , 2010, Bioinform..

[38]  Yiannis Kourmpetis,et al.  Gene Regulatory Networks from Multifactorial Perturbations Using Graphical Lasso: Application to the DREAM4 Challenge , 2010, PloS one.

[39]  P. Geurts,et al.  Inferring Regulatory Networks from Expression Data Using Tree-Based Methods , 2010, PloS one.

[40]  D. Floreano,et al.  Revealing strengths and weaknesses of methods for gene network inference , 2010, Proceedings of the National Academy of Sciences.

[41]  N. D. Clarke,et al.  Correction: Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges , 2010, PLoS ONE.

[42]  Kevin Y. Yip,et al.  Improved Reconstruction of In Silico Gene Regulatory Networks by Integrating Knockout and Perturbation Data , 2010, PloS one.

[43]  T. Schaffter.  Numerical Integration of SDEs: A Short Tutorial , 2010 .

[44]  A. G. de la Fuente,et al.  From Knockouts to Networks: Establishing Direct Cause-Effect Relationships through Graph Analysis , 2010, PloS one.

[45]  Julio Collado-Vides,et al.  RegulonDB version 7.0: transcriptional regulation of Escherichia coli K-12 integrated within genetic sensory response units (Gensor Units) , 2010, Nucleic Acids Res..