Incorporating symbolic domain knowledge into graph neural networks

Our interest is in scientific problems with the following characteristics: (1) Data are naturally represented as graphs; (2) The amount of data available is typically small; and (3) There is significant domain-knowledge, usually expressed in some symbolic form. These kinds of problems have been addressed effectively in the past by Inductive Logic Programming (ILP), by virtue of 2 important characteristics: (a) The use of a representation language that easily captures the relation encoded in graph-structured data, and (b) The inclusion of prior information encoded as domain-specific relations, that can alleviate problems of data scarcity, and construct new relations. Recent advances have seen the emergence of deep neural networks specifically developed for graph-structured data (Graph-based Neural Networks, or GNNs). While GNNs have been shown to be able to handle graph-structured data, less has been done to investigate the inclusion of domain-knowledge. Here we investigate this aspect of GNNs empirically by employing an operation we term "vertex-enrichment" and denote the corresponding GNNs as "VEGNNs". Using over 70 real-world datasets and substantial amounts of symbolic domain-knowledge, we examine the result of vertex-enrichment across 5 different variants of GNNs. Our results provide support for the following: (a) Inclusion of domain-knowledge by vertex-enrichment can significantly improve the performance of a GNN. That is, the performance VEGNNs is significantly better than GNNs across all GNN variants; (b) The inclusion of domain-specific relations constructed using ILP improves the performance of VEGNNs, across all GNN variants. Taken together, the results provide evidence that it is possible to incorporate symbolic domain knowledge into a GNN, and that ILP can play an important role in providing high-level relationships that are not easily discovered by a GNN.

[1]  Yue Gao,et al.  Dynamic Hypergraph Neural Networks , 2019, IJCAI.

[2]  Pierre Baldi,et al.  Graph kernels for chemical informatics , 2005, Neural Networks.

[3]  Kai-Uwe Kühnberger,et al.  Neural-Symbolic Learning and Reasoning: A Survey and Interpretation , 2017, Neuro-Symbolic Artificial Intelligence.

[4]  Ashwin Srinivasan,et al.  Discrete Stochastic Search and Its Application to Feature-Selection for Deep Relational Machines , 2019, ICANN.

[5]  Henk Vandecasteele,et al.  Discovering H-bonding rules in crystals with inductive logic programming. , 2006, Molecular pharmaceutics.

[6]  Christopher H. Bryant,et al.  Functional genomic hypothesis generation and experimentation by a robot scientist , 2004, Nature.

[7]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[8]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[9]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[10]  Igor I. Baskin,et al.  A Neural Device for Searching Direct Correlations between Structures and Properties of Chemical Compounds , 1997, J. Chem. Inf. Comput. Sci..

[11]  Érick Alphonse,et al.  Macro-Operators Revisited in Inductive Logic Programming , 2004, ILP.

[12]  Artur S. d'Avila Garcez,et al.  Fast relational learning using bottom clause propositionalization with artificial neural networks , 2013, Machine Learning.

[13]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[14]  Bernhard Schölkopf,et al.  Learning with Hypergraphs: Clustering, Classification, and Embedding , 2006, NIPS.

[15]  Partha Pratim Talukdar,et al.  HyperGCN: A New Method of Training Graph Convolutional Networks on Hypergraphs , 2018 .

[16]  Luc De Raedt,et al.  ILP turns 20 , 2011, Machine Learning.

[17]  Ugur Kursuncu,et al.  Shades of Knowledge-Infused Learning for Enhancing Deep Learning , 2019, IEEE Internet Computing.

[18]  Ashwin Srinivasan,et al.  Topic Models with Relational Features for Drug Design , 2012, ILP.

[19]  Lorenzo Livi,et al.  Graph Neural Networks With Convolutional ARMA Filters , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  M J Sternberg,et al.  Structure-activity relationships derived by machine learning: the use of atoms and their bond connectivities to predict mutagenicity by inductive logic programming. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[21]  G. Plotkin.  Automatic Methods of Inductive Inference , 1972 .

[22]  Michael Granitzer,et al.  Injecting Semantic Background Knowledge into Neural Networks using Graph Embeddings , 2017, 2017 IEEE 26th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE).

[23]  Jian Pei,et al.  A Survey on Network Embedding , 2017, IEEE Transactions on Knowledge and Data Engineering.

[24]  V. Gold.  Compendium of chemical terminology , 1987 .

[25]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[26]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[27]  Patrick Hoffman,et al.  Data Mining the NCI Cancer Cell Line Compound GI50 Values: Identifying Quinone Subtypes Effective Against Melanoma and Leukemia Cell Classes , 2003, J. Chem. Inf. Comput. Sci..

[28]  Peter A. Flach,et al.  Propositionalization approaches to relational data mining , 2001 .

[29]  Lovekesh Vig,et al.  Large-Scale Assessment of Deep Relational Machines , 2018, ILP.

[30]  Jaewoo Kang,et al.  Self-Attention Graph Pooling , 2019, ICML.

[31]  Pietro Liò,et al.  Towards Sparse Hierarchical Graph Classifiers , 2018, ArXiv.

[32]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[33]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[34]  Artur S. d'Avila Garcez,et al.  Relational Knowledge Extraction from Neural Networks , 2015, CoCo@NIPS.

[35]  Yue Gao,et al.  Hypergraph Neural Networks , 2018, AAAI.

[36]  Saso Dzeroski,et al.  Learning Nonrecursive Definitions of Relations with LINUS , 1991, EWSL.

[37]  Ashwin Srinivasan,et al.  What Kinds of Relational Features Are Useful for Statistical Learning? , 2012, ILP.

[38]  Ashwin Srinivasan,et al.  Feature Construction Using Theory-Guided Sampling and Randomised Search , 2008, ILP.

[39]  Ugur Kursuncu,et al.  Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning , 2020, AAAI Spring Symposium: Combining Machine Learning with Knowledge Engineering.

[40]  Li Guo,et al.  Improving Knowledge Graph Embedding Using Simple Constraints , 2018, ACL.

[41]  Ryoma Sato,et al.  A Survey on The Expressive Power of Graph Neural Networks , 2020, ArXiv.

[42]  Ashwin Srinivasan,et al.  Using ILP to Construct Features for Information Extraction from Semi-structured Text , 2007, ILP.

[43]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[44]  Huma Lodhi,et al.  Deep Relational Machines , 2013, ICONIP.

[45]  Stefan Wrobel,et al.  Macro-Operators in Multirelational Learning: A Search-Space Reduction Technique , 2002, ECML.

[46]  Chengqi Zhang,et al.  Network Representation Learning: A Survey , 2017, IEEE Transactions on Big Data.

[47]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[48]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[49]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[50]  Ashwin Srinivasan,et al.  Feature construction with Inductive Logic Programming: A Study of Quantitative Predictions of Biological Activity Aided by Structural Attributes , 1999, Data Mining and Knowledge Discovery.

[51]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[52]  Artur S. d'Avila Garcez,et al.  The Connectionist Inductive Learning and Logic Programming System , 1999, Applied Intelligence.

[53]  Peter A. Flach,et al.  Comparative Evaluation of Approaches to Propositionalization , 2003, ILP.

[54]  Stephen Muggleton,et al.  Inverse entailment and progol , 1995, New Generation Computing.

[55]  Katherine Yelick,et al.  AI for Science , 2020 .

[56]  Luc De Raedt,et al.  Neuro-Symbolic = Neural + Logical + Probabilistic , 2019, NeSy@IJCAI.

[57]  Artur S. d'Avila Garcez,et al.  Neural Relational Learning Through Semi-Propositionalization of Bottom Clauses , 2015, AAAI Spring Symposia.

[58]  Pablo Barceló,et al.  Logical Expressiveness of Graph Neural Networks , 2019 .