Ontology learning as a use-case for neural-symbolic integration

We argue that the field of neural-symbolic integration is in need of identifying application scenarios for guiding further research. We furthermore argue that ontology learning — as occuring in the context of semantic technologies — provides such an application scenario with potential for success and high impact for neural-symbolic integration. 1 Neural-Symbolic Integration Intelligent systems based on symbolic knowledge processing on the one hand, and on artificial neural networks (also called connectionist systems) on the other, differ substantially. They are both standard approaches to artificial intelligence and it would be very desirable to combine the robust neural networking machinery with symbolic knowledge representation and reasoning paradigms like logic programming in such a way that the strengths of either paradigm will be retained. The importance of these efforts to bridge the gap between the connectionist and symbolic paradigms of Artificial Intelligence has been widely recognised. Since the amount of hybrid data which includes symbolic elements as well as statistical aspects and noise increases dramatically in diverse areas such as bioinformatics or text and web domains, this problem is of particular practical importance. The merging of theory (background knowledge) and data learning (learning from examples) in neural networks has been indicated to provide learning systems that are more effective than purely symbolic and purely connectionist systems, especially when data are noisy and described by real-valued as well as symbolic components. The above results, due also to the massively parallel architecture of neural networks, contributed decisively to the growing interest in developing neural-symbolic systems, i.e. hybrid systems based on neural networks that are capable of learning from examples and background knowledge, and of performing reasoning tasks in a massively parallel fashion. Typically, translation algorithms from a symbolic to a connectionist representation and vice-versa are employed to provide either (i) a neural implementation of a logic, (ii) a logical characterization of a neural system, or (iii) a hybrid system that brings together features from connectionism and symbolic Artificial Intelligence. However, while symbolic knowledge representation is highly recursive and well understood from a declarative point of view, neural networks encode knowledge implicitly in their weights as a result of learning and generalisation from raw data which is usually characterized by simple feature vectors. While significant theoretical progress has recently been made on knowledge representation and reasoning using neural networks on the one side and direct processing of symbolic and structured data with neural methods on the other side, the integration of neural computation and expressive logics such as first order logic is still in its early stages of methodological development. As for knowledge extraction, neural networks have been applied to a variety of real-world problems (e.g. in bioinformatics, engineering, robotics), having been particularly successful when data are noisy, but entirely satisfactory methods for extracting symbolic knowledge from such trained networks are still to be found, and principled problems to ensure the stability and learnability of recursive models currently impose severe restrictions on connectionist systems. In order to advance the state of the art, we believe that it is necessary to look at the biological inspiration for neural-symbolic integration, to use more formal approaches for translating between the connectionist and symbolic paradigms, and to pay more attention to potential application scenarios. We will argue in the following that ontology learning provides such an application scenario with potential for success and high impact. 2 The Need for Use Cases The general motivation for research in the field of neuralsymbolic integration just given arises from conceptual observations on the complementary nature of symbolic and neuralnetwork-based artificial intelligence which we described. This conceptual perspective is sufficient for justifying the mainly foundations-driven lines of research being undertaken in this area so far. However, it appears that this conceptual approach to the study of neural-symbolic integration has now reached an impasse which requires the identification of use cases and application scenarios in order to drive future research. Indeed, the theory of integrated neural-symbolic systems has reached a quite mature state but has not been tested so far on real application data. From the pioneering work by McCulloch and Pitts [22], a number of systems have been developed in the 80s and 90s, including Towell and Shavlik’s KBANN [28], Shastri’s SHRUTI [26], the work by Pinkas [24], Holldobler [17], and d’Avila Garcez et al. [11; 13], to mention a few, and we refer to [8; 12; 15] for comprehensive literature overviews. These systems, however, have been developed for the study of general principles, and are in general not suitable for real data or application scenarios. Nevertheless, these studies provide methods which can be exploited for the development of tools for use cases, and significant progress can now only be expected by developing practical tools out of the fundamental research undertaken in the past. The systems just mentioned — and most of the research on neural-symbolic integration to date — is based on propositional logic or similarly finitistic paradigms. Significantly large and expressible fragments of first order logic are rarely being used because the integration task becomes much harder due to the fact that the underlying language is infinite but shall be encoded using networks with a finite number of nodes [6]. The few approaches known to us for overcoming this problem are work on recursive autoassociative memory, RAAM, initiated by Pollack [25], which concerns the learning of recursive terms over a first-order language, and research based on a proposal by Holldobler et al. [19], spelled out first for the propositional case in [18], and reported also in [16]. It is based on the idea that logic programs can be represented — at least up to subsumption equivalence [21] — by their associated single-step or immediate consequence operators. Such an operator can then be mapped to a function on the real numbers, which can under certain conditions in turn be encoded or approximated e.g. by feedforward networks with sigmoidal activation functions using an approximation theorem due to Funahashi [10]. Despite a number of sophisticated theoretical results building on the latter approach — reported e.g. in [19; 4; 16; 6; 5] —, first-order neural-symbolic integration still appears to be a widely open issue, where advances are very difficult, and it is very hard to judge to date to what extent the theoretical approaches can work in practice. The development of use cases with varying levels of expressive complexity are therefore needed in order to drive the development of methods for neural-symbolic integration beyond propositional logic. 3 Semantic Technologies and Ontology