End to end learning and optimization on graphs

Real-world applications often combine learning and optimization problems on graphs. For instance, our objective may be to cluster the graph in order to detect meaningful communities (or solve other common graph optimization problems such as facility location, maxcut, and so on). However, graphs or related attributes are often only partially observed, introducing learning problems such as link prediction which must be solved prior to optimization. Standard approaches treat learning and optimization entirely separately, while recent machine learning work aims to predict the optimal solution directly from the inputs. Here, we propose an alternative decision-focused learning approach that integrates a differentiable proxy for common graph optimization problems as a layer in learned systems. The main idea is to learn a representation that maps the original optimization problem onto a simpler proxy problem that can be efficiently differentiated through. Experimental results show that our ClusterNet system outperforms both pure end-to-end approaches (that directly predict the optimal solution) and standard approaches that entirely separate learning and optimization. Code for our system is available at this https URL.

[1]  Ambuj Tewari,et al.  Generalization Bounds in the Predict-then-Optimize Framework , 2019, NeurIPS.

[2]  Priya L. Donti,et al.  Task-based End-to-end Model Learning in Stochastic Optimization , 2017, NIPS.

[3]  Justin Domke,et al.  Generic Methods for Optimization-Based Modeling , 2012, AISTATS.

[4]  Nicola Parolini,et al.  Link Prediction in Criminal Networks: A Tool for Criminal Intelligence Analysis , 2016, PloS one.

[5]  Maxim Sviridenko,et al.  Pipage Rounding: A New Method of Constructing Algorithms with Proven Performance Guarantee , 2004, J. Comb. Optim..

[6]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[7]  Steve Gregory,et al.  Detecting community structure in networks using edge prediction methods , 2012, ArXiv.

[8]  Le Song,et al.  2 Common Formulation for Greedy Algorithms on Graphs , 2018 .

[9]  Marc Barthelemy,et al.  Spatial Networks , 2010, Encyclopedia of Social Network Analysis and Mining.

[10]  Adam N. Elmachtoub,et al.  Smart "Predict, then Optimize" , 2017, Manag. Sci..

[11]  J. Zico Kolter,et al.  OptNet: Differentiable Optimization as a Layer in Neural Networks , 2017, ICML.

[12]  Arthur Mensch,et al.  Differentiable Dynamic Programming for Structured Prediction and Attention , 2018, ICML.

[13]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[14]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[15]  Yixin Chen,et al.  Link Prediction Based on Graph Neural Networks , 2018, NeurIPS.

[16]  N. Zdanowicz,et al.  Adolescent health , 2004, The Lancet.

[17]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[18]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[19]  Michalis K. Titsias,et al.  One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities , 2016, NIPS.

[20]  Harri Valpola,et al.  Tagger: Deep Unsupervised Perceptual Grouping , 2016, NIPS.

[21]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[22]  Ronen Basri,et al.  SpectralNet: Spectral Clustering using Deep Neural Networks , 2018, ICLR.

[23]  James Bailey,et al.  An Investigation into Prediction + Optimisation for the Knapsack Problem , 2019, CPAIOR.

[24]  Enhong Chen,et al.  Learning Deep Representations for Graph Clustering , 2014, AAAI.

[25]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[26]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[27]  Andreas Krause,et al.  Differentiable Learning of Submodular Models , 2017, NIPS 2017.

[28]  Xiaochun Cao,et al.  Modularity Based Community Detection with Deep Learning , 2016, IJCAI.

[29]  Marcin Andrychowicz,et al.  Learning to learn by gradient descent by gradient descent , 2016, NIPS.

[30]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[31]  Raquel Urtasun,et al.  Deep Spectral Clustering Learning , 2017, ICML.

[32]  Xin Lu,et al.  Efficient network disintegration under incomplete information: the comic effect of link prediction , 2016, Scientific reports.

[33]  M. Newman,et al.  Finding community structure in networks using the eigenvectors of matrices. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  Samy Bengio,et al.  Neural Combinatorial Optimization with Reinforcement Learning , 2016, ICLR.

[35]  Satish Rao,et al.  Using Max Cut to Enhance Rooted Trees Consistency , 2006, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[36]  Amin Saberi,et al.  Stochastic Submodular Maximization , 2008, WINE.

[37]  M. Newman,et al.  Finding community structure in very large networks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[38]  Martin Grötschel,et al.  An Application of Combinatorial Optimization to Statistical Physics and Circuit Layout Design , 1988, Oper. Res..

[39]  M E J Newman,et al.  Modularity and community structure in networks. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[40]  Milind Tambe,et al.  Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization , 2018, AAAI.

[41]  Max Welling,et al.  Attention, Learn to Solve Routing Problems! , 2018, ICLR.

[42]  Boleslaw K. Szymanski,et al.  Community Detection with Edge Augmentation in Criminal Networks , 2018, 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[43]  Jürgen Schmidhuber,et al.  Neural Expectation Maximization , 2017, NIPS.

[44]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[45]  Teofilo F. GONZALEZ,et al.  Clustering to Minimize the Maximum Intercluster Distance , 1985, Theor. Comput. Sci..

[46]  Max Welling,et al.  Attention Solves Your TSP , 2018, ArXiv.

[47]  Michael J. Cafarella,et al.  Link-Prediction Enhanced Consensus Clustering for Complex Networks , 2015, PloS one.

[48]  Jianping Yin,et al.  Improved Deep Embedded Clustering with Local Structure Preservation , 2017, IJCAI.

[49]  Sune Lehmann,et al.  Link communities reveal multiscale complexity in networks , 2009, Nature.

[50]  Milind Tambe,et al.  Optimizing Network Structure for Preventative Health , 2018, AAMAS.

[51]  Vipin Kumar,et al.  A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs , 1998, SIAM J. Sci. Comput..

[52]  Azalia Mirhoseini,et al.  GAP: Generalizable Approximate Graph Partitioning Framework , 2019, ArXiv.

[53]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[54]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[55]  Ryan A. Rossi,et al.  Learning Role-based Graph Embeddings , 2018, ArXiv.

[56]  Andreas Krause,et al.  Stochastic Submodular Maximization: The Case of Coverage Functions , 2017, NIPS.

[57]  Stanford,et al.  Learning to Discover Social Circles in Ego Networks , 2012 .