Black-box Testing of First-Order Logic Ontologies Using WordNet

Artificial Intelligence aims to provide computer programs with commonsense knowledge to reason about our world. This paper offers a new practical approach towards automated commonsense reasoning with first-order logic (FOL) ontologies. We propose a new black-box testing methodology of FOL SUMO-based ontologies by exploiting WordNet and its mapping into SUMO. Our proposal includes a method for the (semi-)automatic creation of a very large benchmark of competency questions and a procedure for its automated evaluation by using automated theorem provers (ATPs). Applying different quality criteria, our testing proposal enables a successful evaluation of a) the competency of several translations of SUMO into FOL and b) the performance of various automated ATPs. Finally, we also provide a fine-grained and complete analysis of the commonsense reasoning competency of current FOL SUMO-based ontologies.

[1]  Anders Søgaard,et al.  Patrick Blackburn and Johan Bos, Representation and Inference for Natural Language , 2007, Stud Logica.

[2]  M. Minsky.  The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind , 2006 .

[3]  Johan Bos,et al.  Applying automated deduction to natural language understanding , 2009, J. Appl. Log..

[4]  Egoitz Laparra,et al.  Multilingual Central Repository version 3.0 , 2012, LREC.

[5]  Mariano Fernández-López,et al.  Ontological Engineering , 2003, Encyclopedia of Database Systems.

[6]  Christiane Fellbaum,et al.  Putting Semantics into WordNet's "Morphosemantic" Links , 2009, LTC.

[7]  Johan Bos,et al.  Recognising Textual Entailment with Robust Logical Inference , 2005, MLCW.

[8]  Javier Álvez,et al.  Improving the Competency of First-Order Ontologies , 2015, K-CAP.

[9]  Javier Álvez,et al.  Adimen-SUMO: Reengineering an Ontology for First-Order Reasoning , 2012, Int. J. Semantic Web Inf. Syst..

[10]  Ioan Dragan,et al.  vanHelsing: A Fast Proof Checker for Debuggable Compiler Verification , 2015, 2015 17th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC).

[11]  Javier Álvez,et al.  Complete and Consistent Annotation of WordNet using the Top Concept Ontology , 2008, LREC.

[12]  Ernest Davis,et al.  Commonsense reasoning and commonsense knowledge in artificial intelligence , 2015, Commun. ACM.

[13]  Catherine Havasi,et al.  Representing General Relational Knowledge in ConceptNet 5 , 2012, LREC.

[14]  Aldo Gangemi,et al.  Modelling Ontology Evaluation and Validation , 2006, ESWC.

[15]  Javier Álvez,et al.  Cross-checking WordNet and SUMO Using Meronymy , 2018, LREC.

[16]  Josef Ruppenhofer,et al.  FrameNet II: Extended theory and practice , 2006 .

[17]  Adam Pease,et al.  Linking Lixicons and Ontologies: Mapping WordNet to the Suggested Upper Merged Ontology , 2003, IKE.

[18]  Ido Dagan,et al.  Recognizing Textual Entailment: Models and Applications , 2013, Recognizing Textual Entailment: Models and Applications.

[19]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[20]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[21]  Geoff Sutcliffe,et al.  Evaluating general purpose automated theorem proving systems , 2001, Artif. Intell..

[22]  Geoff Sutcliffe,et al.  First Order Reasoning on a Large Ontology , 2007, ESARLT.

[23]  Andrei Voronkov,et al.  First-Order Theorem Proving and Vampire , 2013, CAV.

[24]  Adam Pease,et al.  Towards a standard upper ontology , 2001, FOIS.

[25]  Javier Álvez,et al.  Evaluating Automated Theorem Provers Using Adimen-SUMO , 2016, Vampire@IJCAR.

[26]  M. R. Genesereth,et al.  Knowledge Interchange Format Version 3.0 Reference Manual , 1992, LICS 1992.

[27]  Geoff Sutcliffe,et al.  The development of CASC , 2002, AI Commun..

[28]  Michael S. Bernstein,et al.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations , 2016, International Journal of Computer Vision.

[29]  Geoff Sutcliffe.  The TPTP Problem Library and Associated Infrastructure , 2017, Journal of Automated Reasoning.

[30]  John McCarthy,et al.  Applications of Circumscription to Formalizing Common Sense Knowledge , 1987, NMR.

[31]  Steffen Staab,et al.  International Handbooks on Information Systems , 2013 .

[32]  Lasha Abzianidze,et al.  LangPro: Natural Language Theorem Prover , 2017, EMNLP.

[33]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[34]  Ian Horrocks,et al.  Reasoning Support for Expressive Ontology Languages Using a Theorem Prover , 2006, FoIKS.

[35]  Stephan Schulz,et al.  E - a brainiac theorem prover , 2002, AI Commun..

[36]  Asunción Gómez-Pérez,et al.  Methodological guidelines for reusing general ontologies , 2013, Data Knowl. Eng..

[37]  German Rigau.  Towards Cross-checking WordNet and SUMO Using Meronymy , 2018, GWC.

[38]  B. Hammond.  Ontology , 2004, Lawrence Booth’s Book of Visions.

[39]  C. Fellbaum.  An Electronic Lexical Database , 1998 .

[40]  N. F. Noy,et al.  Ontology Development 101: A Guide to Creating Your First Ontology , 2001 .

[41]  Michael Gruninger,et al.  Methodology for the Design and Evaluation of Ontologies , 1995, IJCAI 1995.

[42]  Egoitz Laparra,et al.  Predicate Matrix: automatically extending the semantic interoperability between predicate resources , 2016, Lang. Resour. Evaluation.

[43]  Andrei Voronkov,et al.  The design and implementation of VAMPIRE , 2002, AI Commun..

[44]  Eneko Agirre,et al.  Interpretable Semantic Textual Similarity: Finding and explaining differences between sentences , 2016, Knowl. Based Syst..

[45]  Geoff Sutcliffe,et al.  The state of CASC , 2006, AI Commun..

[46]  Javier Álvez,et al.  Automatic White-Box Testing of First-Order Logic Ontologies , 2017, J. Log. Comput..

[47]  Adam Pease,et al.  Sigma: An integrated development environment for formal ontology , 2013, AI Commun..

[48]  John McCarthy,et al.  Artificial Intelligence, Logic and Formalizing Common Sense , 1989 .