Exploring Graph Learning for Semi-Supervised Classification Beyond Euclidean Data

Semi-supervised classification on graph-structured data has received increasing attention, where labels are only available for a small subset of data such as social networks and citation networks. This problem is challenging due to the irregularity of graphs. Graph convolutional neural networks (GCN) have been recently proposed to address such kinds of problems, which feed the graph topology into the network to guide operations such as graph convolution. Nevertheless, in most cases where the graphs are not given, they are empirically constructed manually, which tends to be sub-optimal. Hence, we propose Graph Learning Neural Networks (GLNN), which exploits the optimization of graphs (the adjacency matrix in particular) and integrates into the GCN for semi-supervised node classification. Leveraging on spectral graph theory, this essentially combines both graph learning and graph convolution into a unified framework. Specifically, we represent features of social/citation networks as graph signals, and propose the objective of graph learning from the graph-signal prior, sparsity constraint and properties of a valid adjacency matrix via maximum a posteriori estimation. The optimization objective is then integrated into the loss function of the GCN, leading to joint learning of the adjacency matrix and high-level features. Experimental results show that our proposed GLNN outperforms state-of-the-art approaches over widely adopted social network datasets and citation network datasets.

[1]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[2]  Joan Bruna,et al.  Deep Convolutional Networks on Graph-Structured Data , 2015, ArXiv.

[3]  Vincent Gripon,et al.  Graph reconstruction from the observation of diffused signals , 2015, 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[4]  Yue Zhang,et al.  Sentence-State LSTM for Text Representation , 2018, ACL.

[5]  Leon W. Cohen,et al.  Conference Board of the Mathematical Sciences , 1963 .

[6]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[7]  Santiago Segarra,et al.  Network Topology Inference from Spectral Templates , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[8]  Phi Vu Tran,et al.  Learning to Make Predictions on Graphs with Autoencoders , 2018, 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA).

[9]  Daniel A. Spielman,et al.  Fitting a graph to vector data , 2009, ICML '09.

[10]  Oscar C. Au,et al.  Multiresolution Graph Fourier Transform for Compression of Piecewise Smooth Images , 2015, IEEE Transactions on Image Processing.

[11]  Adam J. Rothman,et al.  Sparse permutation invariant covariance estimation , 2008, 0801.4837.

[12]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[13]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[14]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[15]  Ruoyu Li,et al.  Adaptive Graph Convolutional Neural Networks , 2018, AAAI.

[16]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[17]  Hossein Mobahi,et al.  Deep Learning via Semi-supervised Embedding , 2012, Neural Networks: Tricks of the Trade.

[18]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[19]  Fei Wu,et al.  Spatio-Temporal Graph Routing for Skeleton-Based Action Recognition , 2019, AAAI.

[20]  P. Vandergheynst,et al.  Accelerated filtering on graphs using Lanczos method , 2015, 1509.04537.

[21]  Shuicheng Yan,et al.  Semantic Object Parsing with Graph LSTM , 2016, ECCV.

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[24]  Graham Cormode,et al.  Node Classification in Social Networks , 2011, Social Network Data Analytics.

[25]  Nanyun Peng,et al.  Cross-Sentence N-ary Relation Extraction with Graph LSTMs , 2017, TACL.

[26]  Quanzheng Li,et al.  A graph theoretical regression model for brain connectivity learning of Alzheimer'S disease , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[27]  Pradeep Ravikumar,et al.  Sparse inverse covariance matrix estimation using quadratic approximation , 2011, MLSLP.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Mari Ostendorf,et al.  Conversation Modeling on Reddit Using a Graph-Structured LSTM , 2017, TACL.

[30]  Martin J. Wainwright,et al.  Model Selection in Gaussian Graphical Models: High-Dimensional Consistency of l1-regularized MLE , 2008, NIPS.

[31]  Vassilis Kalofolias,et al.  How to Learn a Graph from Smooth Signals , 2016, AISTATS.

[32]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[33]  Antonio Ortega,et al.  Graph Learning from Data under Structural and Laplacian Constraints , 2016, ArXiv.

[34]  Daniel A. Spielman,et al.  Spectral Graph Theory and its Applications , 2007, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[35]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[36]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[37]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[38]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Charles T. Loop,et al.  Point cloud attribute compression with graph transform , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[40]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[41]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[42]  Jaejoon Lee,et al.  Edge-adaptive transforms for efficient depth map coding , 2010, 28th Picture Coding Symposium.

[43]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[44]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[45]  Michael G. Rabbat,et al.  Characterization and Inference of Graph Diffusion Processes From Observations of Stationary Signals , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[46]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[47]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[48]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[49]  Po-Ling Loh,et al.  Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses , 2012, NIPS.

[50]  Lise Getoor,et al.  Entity and Relationship Labeling in Affiliation Networks , 2006 .

[51]  Joshua B. Tenenbaum,et al.  Discovering Structure by Learning Sparse Graphs , 2010 .

[52]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[53]  Xavier Bresson,et al.  CayleyNets: Graph Convolutional Neural Networks With Complex Rational Spectral Filters , 2017, IEEE Transactions on Signal Processing.

[54]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[55]  Sergio Barbarossa,et al.  Graph topology inference based on transform learning , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[56]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[57]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[58]  Ichiro Sakata,et al.  Detecting trends in academic research from a citation network using network representation learning , 2018, PloS one.

[59]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[60]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[61]  Lei Shi,et al.  Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Amin Zheng,et al.  RGCNN: Regularized Graph CNN for Point Cloud Segmentation , 2018, ACM Multimedia.

[63]  Pascal Frossard,et al.  Learning Laplacian Matrix in Smooth Graph Signal Representations , 2014, IEEE Transactions on Signal Processing.

[64]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[65]  Xueqi Cheng,et al.  Graph Wavelet Neural Network , 2019, ICLR.

[66]  Jean-Yves Audibert Optimization for Machine Learning , 1995 .