Framework for an Automated Comparison of Description Logic Reasoners

OWL is an ontology language developed by the W3C, and although initially developed for the Semantic Web, OWL has rapidly become a de facto standard for ontology development in general. The design of OWL was heavily influenced by research in description logics, and the specification includes a formal semantics. One of the goals of this formal approach was to provide interoperability: different OWL reasoners should provide the same results when processing the same ontologies. In this paper we present a system that allows users: (a) to test and compare OWL reasoners using an extensible library of real-life ontologies; (b) to check the “correctness” of the reasoners by comparing the computed class hierarchy; (c) to compare the performance of the reasoners when performing this task; and (d) to use SQL queries to analyse and present the results in any way they see fit.

[1]  Volker Haarslev,et al.  High Performance Reasoning with Very Large Knowledge Bases: A Practical Case Study , 2000, IJCAI.

[2]  Ian Horrocks,et al.  From SHIQ and RDF to OWL: the making of a Web Ontology Language , 2003, J. Web Semant..

[3]  Ian Horrocks,et al.  A Tableaux Decision Procedure for SHOIQ , 2005, IJCAI.

[4]  Zhengxiang Pan.  Benchmarking DL Reasoners Using Realistic Ontologies , 2005, OWLED.

[5]  I. Horrocks,et al.  DL Systems Comparison , 1998 .

[6]  Peter F. Patel-Schneider,et al.  A New General Method to Generate Random Modal Formulae for Testing Decision Procedures , 2003, J. Artif. Intell. Res..

[7]  Ian Horrocks,et al.  Using Vampire to Reason with OWL , 2004, SEMWEB.

[8]  Frank Pfenning,et al.  Logic Programming and Automated Reasoning , 1994, Lecture Notes in Computer Science.

[9]  Roy Dyckhoff.  Automated Reasoning with Analytic Tableaux and Related Methods , 2000, Lecture Notes in Computer Science.

[10]  Larry Wos,et al.  What Is Automated Reasoning? , 1987, J. Autom. Reason..

[11]  Francesco M. Donini,et al.  Design and results of TANCS-00 , 2000 .

[12]  Peter F. Patel-Schneider,et al.  DLP System Description , 1998, Description Logics.

[13]  Peter Crowther,et al.  The DIG Description Logic Interface , 2003, Description Logics.

[14]  Yarden Katz,et al.  Pellet: A practical OWL-DL reasoner , 2007, J. Web Semant..

[15]  Ian Horrocks,et al.  Ontology Reasoning in the SHOQ(D) Description Logic , 2001, IJCAI.

[16]  Ian Horrocks,et al.  Ordering Heuristics for Description Logic Reasoning , 2005, IJCAI.

[17]  Peter F. Patel-Schneider,et al.  FaCT and DLP , 1998, TABLEAUX.

[18]  Jeff Heflin,et al.  An Evaluation of Knowledge Base Systems for Large OWL Datasets , 2004, SEMWEB.

[19]  Ullrich Hustadt,et al.  Using Resolution for Testing Modal Satisfiability and Building Models , 2002, Journal of Automated Reasoning.

[20]  Volker Haarslev,et al.  RACER System Description , 2001, IJCAR.

[21]  John Mylopoulos,et al.  The Semantic Web - ISWC 2003 , 2003, Lecture Notes in Computer Science.

[22]  Francesco M. Donini,et al.  Design and Results of TANCS-2000 Non-classical (Modal) Systems Comparison , 2000, TABLEAUX.

[23]  Ian Horrocks.  The FaCT System , 1998, TABLEAUX.

[24]  Jeffrey M. Bradshaw,et al.  Applying KAoS Services to Ensure Policy Compliance for Semantic Web Services Workflow Composition and Enactment , 2004, SEMWEB.

[25]  Hans van Maaren,et al.  Sat2000: Highlights of Satisfiability Research in the Year 2000 , 2000 .

[26]  Raphael Volz,et al.  Cooking the Semantic Web with the OWL API , 2003, SEMWEB.

[27]  Ian Horrocks.  Benchmark Analysis with FaCT , 2000, TABLEAUX.

[28]  Franz Baader,et al.  An Overview of Tableau Algorithms for Description Logics , 2001, Stud Logica.

[29]  Ian Horrocks,et al.  FaCT++ Description Logic Reasoner: System Description , 2006, IJCAR.

[30]  James A. Hendler,et al.  Swoop: A Web Ontology Editing Browser , 2006, J. Web Semant..

[31]  Ian Horrocks,et al.  Practical Reasoning for Expressive Description Logics , 1999, LPAR.