Powerset Convolutional Neural Networks

We present a novel class of convolutional neural networks (CNNs) for set functions, i.e., data indexed with the powerset of a finite set. The convolutions are derived as linear, shift-equivariant functions for various notions of shifts on set functions. The framework is fundamentally different from graph convolutions based on the Laplacian, as it provides not one but several basic shifts, one for each element in the ground set. Prototypical experiments with several set function classification tasks on synthetic datasets and on datasets derived from real-world hypergraphs demonstrate the potential of our new powerset CNNs.

[1]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[2]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[3]  Claude Berge,et al.  Graphs and Hypergraphs , 2021, Clustering.

[4]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[5]  Ryan O'Donnell,et al.  Learning juntas , 2003, STOC '03.

[6]  Sven de Vries,et al.  Combinatorial Auctions: A Survey , 2003, INFORMS J. Comput..

[7]  Stef Tijs,et al.  Models in Cooperative Game Theory , 2008 .

[8]  Claudio Moraga,et al.  Fourier Analysis on Finite Groups with Applications in Signal Processing and System Design: Stanković/Fourier , 2005 .

[9]  José M. F. Moura,et al.  Algebraic Signal Processing Theory , 2006, ArXiv.

[10]  Andreas Krause,et al.  Near-optimal Observation Selection using Submodular Functions , 2007, AAAI.

[11]  Martin Rötteler,et al.  Algebraic Signal Processing Theory: 2-D Spatial Hexagonal Lattice , 2007, IEEE Trans. Image Process..

[12]  Markus Püschel,et al.  Algebraic Signal Processing Theory: Foundation and 1-D Time , 2008, IEEE Transactions on Signal Processing.

[13]  Kyomin Jung,et al.  Almost Tight Upper Bound for Finding Fourier Coefficients of Bounded Pseudo- Boolean Functions , 2008, COLT.

[14]  Ronald de Wolf,et al.  A Brief Introduction to Fourier Analysis on the Boolean Cube , 2008, Theory Comput..

[15]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[16]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[17]  Maria-Florina Balcan,et al.  Learning submodular functions , 2010, ECML/PKDD.

[18]  Maria-Florina Balcan,et al.  Learning Valuation Functions , 2011, COLT.

[19]  Andrew M. Sutton,et al.  Computing the moments of k-bounded pseudo-Boolean functions over Hamming spheres of arbitrary radius in polynomial time , 2012, Theor. Comput. Sci..

[20]  Andreas Krause,et al.  Learning Fourier Sparse Set Functions , 2012, AISTATS.

[21]  Jelena Kovacevic,et al.  Algebraic Signal Processing Theory: 1-D Nearest Neighbor Models , 2012, IEEE Transactions on Signal Processing.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Tim Roughgarden,et al.  Sketching valuation functions , 2012, SODA.

[24]  T. Sanders,et al.  Analysis of Boolean Functions , 2012, ArXiv.

[25]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[26]  José M. F. Moura,et al.  Discrete Signal Processing on Graphs , 2012, IEEE Transactions on Signal Processing.

[27]  José M. F. Moura,et al.  Discrete Signal Processing on Graphs: Frequency Analysis , 2013, IEEE Transactions on Signal Processing.

[28]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[29]  Andreas Krause,et al.  Submodular Function Maximization , 2014, Tractability.

[30]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[31]  Andreas Krause,et al.  Online Submodular Maximization under a Matroid Constraint with Application to Learning Assignments , 2014, ArXiv.

[32]  Anton Osokin,et al.  Submodular Relaxation for Inference in Markov Random Fields , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[35]  Martin Vetterli,et al.  A Fast Hadamard Transform for Signals With Sublinear Sparsity in the Transform Domain , 2015, IEEE Trans. Inf. Theory.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Yi Lu,et al.  Practical tera-scale Walsh-Hadamard Transform , 2016, 2016 Future Technologies Conference (FTC).

[38]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[40]  Max Welling,et al.  Group Equivariant Convolutional Networks , 2016, ICML.

[41]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[42]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[43]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[44]  Andreas Krause,et al.  Differentiable Learning of Submodular Models , 2017, NIPS 2017.

[45]  Max Welling,et al.  Convolutional Networks for Spherical Signals , 2017, ArXiv.

[46]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[47]  Barnabás Póczos,et al.  Equivariance Through Parameter-Sharing , 2017, ICML.

[48]  Nathan D. Cahill,et al.  Robust Spatial Filtering With Graph Convolutional Neural Networks , 2017, IEEE Journal of Selected Topics in Signal Processing.

[49]  Wenruo Bai,et al.  Deep Submodular Functions , 2017, ArXiv.

[50]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[51]  Leonidas J. Guibas,et al.  SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[53]  Pierre Vandergheynst,et al.  Graph Signal Processing: Overview, Challenges, and Applications , 2017, Proceedings of the IEEE.

[54]  Jan Dirk Wegner,et al.  Inference, Learning and Attention Mechanisms that Exploit and Preserve Sparsity in CNNs , 2018, GCPR.

[55]  Markus Püschel A Discrete Signal Processing Framework for Set Functions , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[56]  Nikos Komodakis,et al.  GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders , 2018, ICANN.

[57]  Jon M. Kleinberg,et al.  Simplicial closure and higher-order link prediction , 2018, Proceedings of the National Academy of Sciences.

[58]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[59]  Andreas Krause,et al.  Differentiable Submodular Maximization , 2018, IJCAI.

[60]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[61]  Markus Püschel,et al.  A Discrete Signal Processing Framework for Meet/join Lattices with Applications to Hypergraphs and Trees , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[62]  Chris Wendler,et al.  Sampling Signals On Meet/Join Lattices , 2019, 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[63]  Knut Hüper,et al.  The Discrete Cosine Transform on Triangles , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[64]  Ryan L. Murphy,et al.  Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs , 2018, ICLR.

[65]  Michael A. Osborne,et al.  On the Limitations of Representing Functions on Sets , 2019, ICML.

[66]  Priya L. Donti,et al.  SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver , 2019, ICML.

[67]  David L. Dill,et al.  Learning a SAT Solver from Single-Bit Supervision , 2018, ICLR.