The Graph Neural Network Model

Many underlying relationships among data in several areas of science and engineering, e.g., computer vision, molecular chemistry, molecular biology, pattern recognition, and data mining, can be represented in terms of graphs. In this paper, we propose a new neural network model, called graph neural network (GNN) model, that extends existing neural network methods for processing the data represented in graph domains. This GNN model, which can directly process most of the practically useful types of graphs, e.g., acyclic, cyclic, directed, and undirected, implements a function tau(G,n) isin IRm that maps a graph G and one of its nodes n into an m-dimensional Euclidean space. A supervised learning algorithm is derived to estimate the parameters of the proposed GNN model. The computational cost of the proposed algorithm is also considered. Some experimental results are shown to validate the proposed learning algorithm, and to demonstrate its generalization capabilities.

[1]  M. J. D. Powell,et al.  An efficient method for finding the minimum of a function of several variables without calculating derivatives , 1964, Comput. J..

[2]  Theodosios Pavlidis,et al.  Structural pattern recognition , 1977 .

[3]  Alexander Graham,et al.  Kronecker Products and Matrix Calculus: With Applications , 1981 .

[4]  Eugene Seneta Markov Chains and Finite Stochastic Matrices , 1981 .

[5]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Valerie Isham,et al.  Non‐Negative Matrices and Markov Chains , 1983 .

[7]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[8]  W. Rudin Real and complex analysis, 3rd ed. , 1987 .

[9]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[10]  Lin-Bao Yang,et al.  Cellular neural networks: theory , 1988 .

[11]  Leon O. Chua,et al.  Cellular neural networks: applications , 1988 .

[12]  L.O. Chua,et al.  Cellular neural networks , 1993, 1988., IEEE International Symposium on Circuits and Systems.

[13]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[14]  L. B. Almeida A learning rule for asynchronous perceptrons with feedback in a combinatorial environment , 1990 .

[15]  Richard S. Sutton,et al.  Neural networks for control , 1990 .

[16]  A. Debnath,et al.  Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity. , 1991, Journal of medicinal chemistry.

[17]  P. Kaluzny Counting stable equilibria of cellular neural networks-A graph theoretic approach , 1992, CNNA '92 Proceedings Second International Workshop on Cellular Neural Networks and Their Applications.

[18]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[19]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[20]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[21]  R. Mike Cameron-Jones,et al.  FOIL: A Midterm Report , 1993, ECML.

[22]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[23]  G. Kane Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[24]  Ashwin Srinivasan,et al.  Mutagenesis: ILP experiments in a non-determinate biological domain , 1994 .

[25]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[26]  Finn Verner Jensen,et al.  Introduction to Bayesian Networks , 2008, Innovations in Bayesian Networks.

[27]  J. Ross Quinlan,et al.  Boosting First-Order Learning , 1996, ALT.

[28]  Christoph Goller,et al.  Inductive Learning in Symbolic Domains Using Structure-Driven Recurrent Neural Networks , 1996, KI.

[29]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[30]  Ah Chung Tsoi,et al.  Gradient Based Learning Methods , 1997, Summer School on Neural Networks.

[31]  Giovanni Soda,et al.  Logo Recognition by Recursive Neural Networks , 1997, GREC.

[32]  Luc De Raedt,et al.  Using Logical Decision Trees for Clustering , 1997, ILP.

[33]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[34]  Marco Gori,et al.  Adaptive processing of sequences and data structures : International Summer School on Neural Networks "E.R. Caianiello", Vietri sul Mare, Salerno, Italy, September 6-13, 1997, tutorial lectures , 1998 .

[35]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[36]  Hector Garcia-Molina,et al.  Efficient Crawling Through URL Ordering , 1998, Comput. Networks.

[37]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[38]  Ah Chung Tsoi,et al.  Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results , 1998, Neural Networks.

[39]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[40]  Marco Gori,et al.  Adaptive Processing of Sequences and Data Structures , 1998, Lecture Notes in Computer Science.

[41]  C. Goller,et al.  Relating Chemical Structure to Activity: An Application of the Neural Folding Architecture , 1998 .

[42]  Luc De Raedt,et al.  Relational Learning and Inductive Logic Programming Made Easy Abstract of Tutorial , 1999, PKDD.

[43]  M. KleinbergJon Authoritative sources in a hyperlinked environment , 1999 .

[44]  Andrew McCallum,et al.  Learning to Create Customized Authority Lists , 2000, ICML.

[45]  J. Nazuno Haykin, Simon. Neural networks: A comprehensive foundation, Prentice Hall, Inc. Segunda Edición, 1999 , 2000 .

[46]  Franco Scarselli,et al.  Learning User Profiles in NAUTILUS , 2000, AH.

[47]  Franco Scarselli,et al.  Processing directed acyclic graphs with recursive neural networks , 2001, IEEE Trans. Neural Networks.

[48]  W. A. Kirk,et al.  An Introduction to Metric Spaces and Fixed Point Theory , 2001 .

[49]  Alessio Micheli,et al.  Analysis of the Internal Representations Developed by Neural Networks for Structures Applied to Quantitative Structure-Activity Relationship Studies of Benzodiazepines , 2001, J. Chem. Inf. Comput. Sci..

[50]  Edwin H. Blake,et al.  A Graphical Representation of the State Spaces of Hierarchical Level-of-Detail Scene Descriptions , 2001, IEEE Trans. Vis. Comput. Graph..

[51]  Luc De Raedt,et al.  Feature Construction with Version Spaces for Biochemical Applications , 2001, ICML.

[52]  Mathias Kirsten,et al.  Multirelational distance based clustering , 2001 .

[53]  Michael Collins,et al.  Convolution Kernels for Natural Language , 2001, NIPS.

[54]  Ah Chung Tsoi,et al.  A Supervised Self-Organizing Map for Structured Data , 2001, WSOM.

[55]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[56]  Jan Ramon Thesis: clustering and instance based learning in first order logic , 2002 .

[57]  Jan Ramon,et al.  Clustering and instance based learning in first order logic , 2002, AI Communications.

[58]  Marco Gori,et al.  Web page scoring systems for horizontal and vertical search , 2002, WWW.

[59]  John D. Lafferty,et al.  Diffusion Kernels on Graphs and Other Discrete Input Spaces , 2002, ICML.

[60]  Marco Gori,et al.  Recursive Neural Networks Applied to Discourse Representation Theory , 2002, ICANN.

[61]  Risi Kondor,et al.  Diffusion kernels on graphs and other discrete structures , 2002, ICML 2002.

[62]  Taher H. Haveliwala Topic-sensitive PageRank , 2002, IEEE Trans. Knowl. Data Eng..

[63]  Ah Chung Tsoi,et al.  Adaptive ranking of web pages , 2003, WWW '03.

[64]  Franco Scarselli,et al.  Face Spotting in Color Images using Recursive Neural Networks , 2003 .

[65]  Pierre Baldi,et al.  The Principled Design of Large-Scale Recursive Neural Network Architectures--DAG-RNNs and the Protein Structure Prediction Problem , 2003, J. Mach. Learn. Res..

[66]  Ah Chung Tsoi,et al.  A self-organizing map for adaptive processing of structured data , 2003, IEEE Trans. Neural Networks.

[67]  Hisashi Kashima,et al.  Marginalized Kernels Between Labeled Graphs , 2003, ICML.

[68]  Emiel Krahmer,et al.  Graph-Based Generation of Referring Expressions , 2003, CL.

[69]  Jennifer Widom,et al.  Scaling personalized web search , 2003, WWW '03.

[70]  Thomas Gärtner,et al.  A survey of kernels for structured data , 2003, SKDD.

[71]  Jun Suzuki,et al.  Kernels for Structured Natural Language Data , 2003, NIPS.

[72]  Peter A. Flach,et al.  Comparative Evaluation of Approaches to Propositionalization , 2003, ILP.

[73]  Mario Vento,et al.  Graph matching applications in pattern recognition and image processing , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[74]  Taher H. Haveliwala Topic-Sensitive PageRank: A Context-Sensitive Ranking Algorithm for Web Search , 2003, IEEE Trans. Knowl. Data Eng..

[75]  Christian S. Collberg,et al.  A system for graph-based visualization of the evolution of software , 2003, SoftVis '03.

[76]  Thomas Villmann,et al.  Supervised relevance neural gas and unified maximum separability analysis for classification of mass spectrometric data , 2004, 2004 International Conference on Machine Learning and Applications, 2004. Proceedings..

[77]  Thomas Gärtner,et al.  Kernels and Distances for Structured Data , 2004, Machine Learning.

[78]  Luciano Baresi,et al.  Tutorial Introduction to Graph Transformation: A Software Engineering Perspective , 2002, ICGT.

[79]  Ah Chung Tsoi,et al.  Computing personalized pageranks , 2004, WWW Alt. '04.

[80]  Tatsuya Akutsu,et al.  Extensions of marginalized graph kernels , 2004, ICML.

[81]  Jun Suzuki,et al.  Convolution Kernels with Feature Selection for Natural Language Processing Tasks , 2004, ACL.

[82]  Barbara Hammer,et al.  Neural methods for non-standard data , 2004, ESANN.

[83]  Mario Vento,et al.  Thirty Years Of Graph Matching In Pattern Recognition , 2004, Int. J. Pattern Recognit. Artif. Intell..

[84]  Stephen Muggleton,et al.  Machine Learning for Systems Biology , 2005, ILP.

[85]  Franco Scarselli,et al.  Recursive neural networks for processing graphs with labelled edges: theory and applications , 2005, Neural Networks.

[86]  C. Merkwirth,et al.  Pattern recognition using finite-iteration cellular systems , 2005, 2005 9th International Workshop on Cellular Neural Networks and Their Applications.

[87]  Ah Chung Tsoi,et al.  Graph neural networks for ranking Web pages , 2005, The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI'05).

[88]  Ah Chung Tsoi,et al.  A supervised training algorithm for self-organizing maps for structures , 2005, Pattern Recognit. Lett..

[89]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[90]  Franco Scarselli,et al.  Recursive neural networks learn to localize faces , 2005, Pattern Recognit. Lett..

[91]  Soumen Chakrabarti,et al.  Learning to rank networked entities , 2006, KDD '06.

[92]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[93]  Franco Scarselli,et al.  Two connectionist models for graph processing: An experimental comparison on relational data , 2006 .

[94]  Ah Chung Tsoi,et al.  Computing customized page ranks , 2006, TOIT.

[95]  Franco Scarselli,et al.  A Comparison between Recursive Neural Networks and Graph Neural Networks , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[96]  Ah Chung Tsoi,et al.  Document Mining Using Graph Neural Network , 2006, INEX.

[97]  Franco Scarselli,et al.  Graph Neural Networks for Object Localization , 2006, ECAI.

[98]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[99]  M. Hilario,et al.  Matching Based Kernels for Labeled Graphs , 2006 .

[100]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[101]  Ben Taskar,et al.  Introduction to Statistical Relational Learning (Adaptive Computation and Machine Learning) , 2007 .

[102]  E. Seneta Non-negative Matrices and Markov Chains , 2008 .

[103]  Luc De Raedt,et al.  Logical and relational learning , 2008, Cognitive Technologies.

[104]  Lakhmi C. Jain,et al.  Introduction to Bayesian Networks , 2008 .

[105]  Luc De Raedt,et al.  Logical and Relational Learning: From ILP to MRDM (Cognitive Technologies) , 2008 .

[106]  Ah Chung Tsoi,et al.  Computational Capabilities of Graph Neural Networks , 2009, IEEE Transactions on Neural Networks.

[107]  R. Cooke Real and Complex Analysis , 2011 .

[108]  H. Bunke Graph Matching : Theoretical Foundations , Algorithms , and Applications , 2022 .