Learning Symbolic Inferences with Neural Networks

Learning Symbolic Inferences with Neural Networks Helmar Gust (hgust@uos.de) Institute of Cognitive Science, University of Osnabr¨uck Katharinenstr. 24, 49069 Osnabr¨uck, Germany Kai-Uwe K uhnberger (kkuehnbe@uos.de) Institute of Cognitive Science, University of Osnabr¨uck Katharinenstr. 24, 49069 Osnabr¨uck, Germany Abstract well-known that classical logical connectives like conjunc- tion, disjunction, or negation can be represented by neural networks (Rojas, 1996). Furthermore it is known that every Boolean function can be learned by a neural network (Stein- bach & Kohut, 2002). Although it is therefore straightfor- ward to represent propositional logic with neural networks, this is not true for FOL. The corresponding problem, usually called the variable-binding problem, is caused by the usage of quantifiers ∀ and ∃, which are binding variables that occur at different positions in one and the same formula. It is there- fore no surprise that there are a number of attempts to solve this problem of neural networks: Examples for such attempts are sign propagation (Lange & Dyer, 1989), dynamic localist representations (Barnden, 1989), tensor product representa- tions (Smolensky, 1990), or holographic reduced representa- tions (Plate, 1994). Unfortunately these accounts have certain non-trivial side-effects. Whereas sign propagation as well as dynamic localist representations lack the ability of learning, the tensor product representation results in an exponentially increasing number of elements to represent variable bindings, only to mention some of the problems. With respect to the inference problem of connectionist net- works the number of proposed solutions is rather small and relatively new. An attempt is Hitzler, H¨olldobler & Seda (2004) in which a logical deduction operator is approximated by a neural network and the fixpoint of such an operator pro- vides the semantics of a logical theory. Another approach is Healy & Caudell (2004) where category theoretic meth- ods are assigned to neural constructions. In D’Avila Garcez, Broda & Gabbay (2002), tractable fragments of predicate logic are learned by connectionist networks. Finally in Gust & K¨uhnberger (2004), a procedure is given how to trans- late predicate logic into variable-free logic that can be used as input for a neural network. To the knowledge of the au- thors, the latter account is the only one that does not require hard-wired networks designed for modeling a particular the- ory. Rather one network topology can be used for arbitrary first-order theories. We will apply the account presented in Gust & K¨uhnberger (2004) to model first-order inferences of neural networks and to discuss issues relevant for cognitive science. The paper has the following structure: First, we will sketch the basic ideas of variable-free first-order logic using a rep- resentation of FOL induced by category-theoretic means in a topos. Second, we will present the general architecture of the system, the structure of the neural network to code variable- free logic, and the neural modeling of inferences processes. In this paper, we will present a theory of representing sym- bolic inferences of first-order logic with neural networks. The approach transfers first-order logical formulas into a variable- free representation (in a topos) that can be used to generate homogeneous equations functioning as input data for a neural network. An evaluation of the results will be presented and some cognitive implications will be discussed. Keywords: Neural Networks; First-Order Inferences; Neural- Symbolic Logic. Introduction The syntactic structure of formulas of classical first-order logic (FOL) is recursively defined. Therefore it is possible to construct new formulas using given ones by applying a recursion principle. Similarly semantic values of (complex) formulas can be computed by the interpretation of the corre- sponding parts (Hodges, 1997). Consider, for example, the following formula: ∀x : human(x) → mortal(x) The semantics of the complex formula is based on the seman- tics of the subexpressions human(x), mortal(x), and the impli- cation → connecting these subexpressions. Clearly a problem arises because of the presence of the quantifier ∀ and the vari- able x. Nevertheless it is assumed that a compositionality principle allows to compute the meaning of the complex for- mula using the meaning of the corresponding subformulas. 1 On the other side, it is assumed that neural networks are non-compositional on a principal basis making it difficult to represent complex data structures like lists, trees, tables, for- mulas etc. Two aspects can be distinguished: The represen- tation problem (Barnden, 1989) and the inference problem (Shastri & Ajjanagadde, 1990). The first problem states that, if at all, complex data structures can only implicitly be used and the representation of structured objects is a non-trivial challenge for connectionist networks. The second problem tries to model inferences of logical systems with neural ac- counts. In this paper, our primary aim is the second problem. A certain endeavor has been invested to solve the repre- sentation problem as well as the inference problem. It is In classical logic, variables are not only used to express quantifi- cation but also to syntactically mark multiple occurrences of terms. Variable management is usually considered as a problematic issue in logic. In particular, the problem arises that algorithms have certain difficulties with quantified variables: The non-decidability of FOL is a direct consequence of this fact.