Learning to Learn Concept Descriptions

The goal of automatically encoding natural language text into some formal representation has been pursued in the field of Knowledge Engineering to support the construction of Formal Ontologies. Many \SOA{} methods have been proposed for the automatic extraction of lightweight Ontologies and to populate them. Only few have tackled the challenge of extracting expressive axioms that formalize the possibly complex semantics of ontological concepts. In this thesis, we address the problem of encoding a natural language sentence expressing the description of a concept into a corresponding Description Logic axiom. In our approach, the encoding happens through a syntactic transformation, so that all the extralogical symbols in the formula are words actually occurring in the input sentence. We followed the recent advances in the field of Deep Learning in order to design suitable Neural Network architectures capable to learn by examples how to perform this transformation. Since no pre-existing dataset was available to adequately train Neural Networks for this task, we designed a data generation pipeline to produce datasets to train and evaluate the architectures proposed in this thesis. These datasets provide therefore a first reference corpus for the task of learning concept description axioms from text via Machine Learning techniques, and are now available for the Knowledge Engineering community to fill the pre-existing lack of data. During our evaluation, we assessed some key characteristics of the approach we propose. First, we evaluated the capability of the trained models to generalize over the syntactic structures used in the expression of concept descriptions, together with the tolerance to unknown words. The importance of these characteristics is due to the fact that Machine Learning systems are trained on a statistical sample of the problem space, and they have to learn to generalize over this sample in order to process new inputs. In particular, in our scenario, even an extremely large training set is not able to include all the possible ways a human can express the definition of a concept. At the same time, part of the human vocabulary is likely to fall out of the training set. Thus, testing these generalization capabilities and the tolerance to unknown words is crucial to evaluate the effectiveness of the model. Second, we evaluated the improvement in the performance of the model when it is incrementally trained with additional training examples. This is also a pivotal characteristic of our approach, since Machine Learning-based systems are typically supposed to continuously evolve and improve, on the long term, through iterative repetitions of training set enlargements and training process runs. Therefore, a valuable model has to show the ability to improve its performance when new training examples are added to the training set. To the best of our knowledge, this work represents the first assessment of an approach to the problem of encoding expressive concept descriptions from text that is entirely Machine Learning-based and is trained in a end-to-end fashion starting from raw text. In detail, this thesis proposes the first two Neural Network architectures in literature to solve the problem together with their evaluation with respect to the above pivotal characteristics, and a first dataset generation pipeline together with concrete datasets.

[1]  Aldo Gangemi,et al.  Knowledge Extraction Based on Discourse Representation Theory and Linguistic Frames , 2012, EKAW.

[2]  Yuh-Min Chen,et al.  Enhancement of domain ontology construction using a crystallizing approach , 2011, Expert Syst. Appl..

[3]  Danqi Chen,et al.  A Fast and Accurate Dependency Parser using Neural Networks , 2014, EMNLP.

[4]  Dieter Fensel,et al.  Knowledge Engineering: Principles and Methods , 1998, Data Knowl. Eng..

[5]  Ossama Emam,et al.  Unsupervised Information Extraction Approach Using Graph Mutual Reinforcement , 2006, EMNLP.

[6]  Claire Gardent,et al.  Mapping Natural Language to Description Logic , 2017, ESWC.

[7]  Matthew D. Zeiler ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[8]  Chris Dyer,et al.  Semantic Parsing with Semi-Supervised Sequential Autoencoders , 2016, EMNLP.

[9]  Alex Graves,et al.  Sequence Transduction with Recurrent Neural Networks , 2012, ArXiv.

[10]  Steffen Staab,et al.  What Is an Ontology? , 2009, Handbook on Ontologies.

[11]  Johanna Völker,et al.  Acquisition of OWL DL Axioms from Lexical Resources , 2007, ESWC.

[12]  D. Floreano,et al.  Manuale sulle Reti Neurali , 2002 .

[13]  Christopher Potts,et al.  Recursive Neural Networks for Learning Logical Semantics , 2014, ArXiv.

[14]  Daniel S. Weld,et al.  Open Information Extraction Using Wikipedia , 2010, ACL.

[15]  Nicola Guarino,et al.  Evaluating ontological decisions with OntoClean , 2002, CACM.

[16]  Dunja Mladenic,et al.  OntoGen: Semi-automatic Ontology Editor , 2007, HCI.

[17]  Ralph Grishman,et al.  Discovering Relations among Named Entities from Large Corpora , 2004, ACL.

[18]  Mirella Lapata,et al.  Language to Logical Form with Neural Attention , 2016, ACL.

[19]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[20]  Satoshi Sekine,et al.  Preemptive Information Extraction using Unrestricted Relation Discovery , 2006, NAACL.

[21]  Andrew Y. Ng,et al.  Parsing Natural Scenes and Natural Language with Recursive Neural Networks , 2011, ICML.

[22]  Oren Etzioni,et al.  Open Information Extraction from the Web , 2007, CACM.

[23]  Paola Velardi,et al.  Learning Word-Class Lattices for Definition and Hypernym Extraction , 2010, ACL.

[24]  Stefano Faralli,et al.  OntoLearn Reloaded: A Graph-Based Algorithm for Taxonomy Induction , 2013, CL.

[25]  Sergey Brin,et al.  Extracting Patterns and Relations from the World Wide Web , 1998, WebDB.

[26]  Aron Culotta,et al.  Dependency Tree Kernels for Relation Extraction , 2004, ACL.

[27]  Daniel Jurafsky,et al.  Semantic Taxonomy Induction from Heterogenous Evidence , 2006, ACL.

[28]  Pari Delir Haghighi,et al.  CFinder: An intelligent key concept finder from text for ontology development , 2014, Expert Syst. Appl..

[29]  James H. Martin,et al.  Speech and Language Processing, 2nd Edition , 2008 .

[30]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[31]  Carl Gutwin,et al.  KEA: practical automatic keyphrase extraction , 1999, DL '99.

[32]  Stefanie N. Lindstaedt,et al.  MoKi: The Enterprise Modelling Wiki , 2009, ESWC.

[33]  Paul Buitelaar,et al.  RelExt: A Tool for Relation Extraction from Text in Ontology Extension , 2005, SEMWEB.

[34]  Gert Smolka,et al.  Attributive Concept Descriptions with Complements , 1991, Artif. Intell..

[35]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[36]  Luis Gravano,et al.  Snowball: extracting relations from large plain-text collections , 2000, DL '00.

[37]  Phil Blunsom,et al.  Learning to Transduce with Unbounded Memory , 2015, NIPS.

[38]  Ronald J. Brachman,et al.  An overview of the KL-ONE Knowledge Representation System , 1985 .

[39]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[40]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[41]  Ahmed A. Rafea,et al.  KP-Miner: A keyphrase extraction system for English and Arabic documents , 2009, Inf. Syst..

[42]  Dmitry Zelenko,et al.  Kernel methods for relation extraction , 2003 .

[43]  Frank van Harmelen,et al.  Web Ontology Language: OWL , 2004, Handbook on Ontologies.

[44]  Werner Nutt,et al.  Basic Description Logics , 2003, Description Logic Handbook.

[45]  Daniel Jurafsky,et al.  Distant supervision for relation extraction without labeled data , 2009, ACL.

[46]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[47]  Kaarel Kaljurand,et al.  Attempto Controlled English for Knowledge Representation , 2008, Reasoning Web.

[48]  Daniel Jurafsky,et al.  Learning Syntactic Patterns for Automatic Hypernym Discovery , 2004, NIPS.

[49]  Honglak Lee,et al.  Unsupervised learning of hierarchical representations with convolutional deep belief networks , 2011, Commun. ACM.

[50]  Oren Etzioni,et al.  Open Information Extraction: The Second Generation , 2011, IJCAI.

[51]  Kam-Fai Wong,et al.  Towards Neural Network-based Reasoning , 2015, ArXiv.

[52]  Mirella Lapata,et al.  Taxonomy Induction Using Hierarchical Random Graphs , 2012, NAACL.

[53]  Brian Davis,et al.  Extracting Semantic Knowledge from Unstructured Text Using Embedded Controlled Language , 2016, 2016 IEEE Tenth International Conference on Semantic Computing (ICSC).

[54]  Emanuele Pianta,et al.  KX: A Flexible System for Keyphrase eXtraction , 2010, *SEMEVAL.

[55]  Nils J. Nilsson,et al.  The Quest for Artificial Intelligence , 2009 .

[56]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[57]  Ah-Hwee Tan,et al.  Mining ontological knowledge from domain-specific text documents , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[58]  Geoffrey E. Hinton,et al.  Grammar as a Foreign Language , 2014, NIPS.

[59]  Chiara Ghidini,et al.  Ontology Learning in the Deep , 2016, EKAW.

[60]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[61]  Dunja Mladenic,et al.  OntoPlus: Text-driven ontology extension using ontology content, structure and co-occurrence information , 2011, Knowl. Based Syst..

[62]  Edgar Tello-Leal,et al.  An Approach for Learning Expressive Ontologies in Medical Domain , 2015, Journal of Medical Systems.

[63]  David Sánchez,et al.  Discovery of Relation Axioms from the Web , 2010, KSEM.

[64]  Asunción Gómez-Pérez,et al.  The NeOn Methodology for Ontology Engineering , 2012, Ontology Engineering in a Networked World.

[65]  Emanuele Pianta,et al.  Boosting Collaborative Ontology Building with Key-Concept Extraction , 2011, 2011 IEEE Fifth International Conference on Semantic Computing.

[66]  Alifah Syamsiyah,et al.  A hybrid approach to learn description logic based biomedical ontology from texts , 2014 .

[67]  Marek Hatala,et al.  Linguistic Patterns for Information Extraction in OntoCmaps , 2012, WOP.

[68]  P. Sreenivasa Kumar,et al.  Extracting Ontological Knowledge from Textual Descriptions through Grammar-based Transformation , 2017, K-CAP.

[69]  Bo Zhang,et al.  StatSnowball: a statistical approach to extracting entity relationships , 2009, WWW '09.

[70]  Mauro Dragoni,et al.  A Neural Word Embeddings Approach for Multi-Domain Sentiment Analysis , 2017, IEEE Transactions on Affective Computing.

[71]  Estevam R. Hruschka,et al.  Discovering Relations between Noun Categories , 2011, EMNLP.

[72]  Johan Bos,et al.  Wide-Coverage Semantic Analysis with Boxer , 2008, STEP.

[73]  Marti A. Hearst Automatic Acquisition of Hyponyms from Large Text Corpora , 1992, COLING.

[74]  Gerhard Weikum,et al.  PATTY: A Taxonomy of Relational Patterns with Semantic Types , 2012, EMNLP.

[75]  Roberto Navigli,et al.  Integrating Syntactic and Semantic Analysis into the Open Information Extraction Paradigm , 2013, IJCAI.

[76]  Diego Calvanese,et al.  The Description Logic Handbook: Theory, Implementation, and Applications , 2003, Description Logic Handbook.

[77]  Steffen Staab,et al.  Ontology Learning , 2004, Encyclopedia of Machine Learning and Data Mining.

[78]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[79]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[80]  Franz Baader,et al.  Formalizing biomedical concepts from textual definitions , 2015, J. Biomed. Semant..

[81]  Henry H. Rueter,et al.  Extracting expertise from experts: Methods for knowledge acquisition , 1987 .

[82]  Ahmed A. Rafea,et al.  TextOntoEx: Automatic ontology construction from natural English text , 2008, Expert Syst. Appl..

[83]  Steffen Staab,et al.  Ontology Engineering Methodology , 2009, Handbook on Ontologies.

[84]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[85]  Bowen Zhou,et al.  Pointing the Unknown Words , 2016, ACL.

[86]  Wojciech Zaremba,et al.  Learning to Execute , 2014, ArXiv.

[87]  Zornitsa Kozareva,et al.  A Semi-Supervised Method to Learn and Construct Taxonomies Using the Web , 2010, EMNLP.

[88]  Razvan C. Bunescu,et al.  A Shortest Path Dependency Kernel for Relation Extraction , 2005, HLT.