LinSATNet: The Positive Linear Satisfiability Neural Networks

Encoding constraints into neural networks is attractive. This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfi-ability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions. We further theoretically characterize the convergence property of the Sinkhorn algorithm for multiple marginals, and the underlying formulation is also derived. In contrast to the sequential decision e.g. reinforcement learning-based solvers, we showcase our technique in solving constrained (specifically sat-isfiability) problems by one-shot neural networks, including i) a neural routing solver learned without supervision of optimal solutions; ii) a partial graph matching network handling graphs with un-matchable outliers on both sides; iii) a predictive network for financial portfolios with continuous constraints. To our knowledge, there exists no one-shot neural solver for these scenarios when they are formulated as satisfiability problems.

[1]  Junchi Yan,et al.  HardSATGEN: Understanding the Difficulty of Hard SAT Formula Generation and A Strong Structure-Hardness-Aware Baseline , 2023, Knowledge Discovery and Data Mining.

[2]  Junchi Yan,et al.  Machine Learning Methods in Solving the Boolean Satisfiability Problem , 2022, Machine Intelligence Research.

[3]  Youngjune Gwon,et al.  Matrix Encoding Networks for Neural Combinatorial Optimization , 2021, NeurIPS.

[4]  K. Freivalds,et al.  Goal-Aware Neural SAT Solver , 2021, 2022 International Joint Conference on Neural Networks (IJCNN).

[5]  Junchi Yan,et al.  A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphs , 2021, NeurIPS.

[6]  G. Martius,et al.  CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraints , 2021, ICML.

[7]  R. Kwon,et al.  Integrating prediction in mean-variance portfolio optimization , 2021, SSRN Electronic Journal.

[8]  Junchi Yan,et al.  Revocable Deep Reinforcement Learning with Affinity Regularization for Outlier-Robust Graph Matching , 2020, ICLR.

[9]  Junchi Yan,et al.  Combinatorial Learning of Graph Edit Distance via Dynamic Embedding , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Qi Zhang,et al.  Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting , 2020, NeurIPS.

[11]  Stephen J. Roberts,et al.  Deep Learning for Portfolio Optimization , 2020, The Journal of Financial Data Science.

[12]  Junchi Yan,et al.  Learning for Graph Matching and Related Combinatorial Optimization Problems , 2020, IJCAI.

[13]  Junchi Yan,et al.  Combinatorial Learning of Robust Deep Graph Matching: An Embedding Based Approach , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Andreas Loukas,et al.  Erdos Goes Neural: an Unsupervised Learning Framework for Combinatorial Optimization on Graphs , 2020, NeurIPS.

[15]  Baoxin Li,et al.  Learning deep graph matching with channel-independent embedding and Hungarian attention , 2020, ICLR.

[16]  G. Martius,et al.  Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers , 2020, ECCV.

[17]  G. Martius,et al.  Differentiation of Blackbox Combinatorial Solvers , 2019, ICLR.

[18]  Junchi Yan,et al.  Neural Graph Matching Network: Learning Lawler’s Quadratic Assignment Problem With Extension to Hypergraph and Multiple-Graph Matching , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Stephen P. Boyd,et al.  Differentiable Convex Optimization Layers , 2019, NeurIPS.

[20]  Xavier Bresson,et al.  An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem , 2019, ArXiv.

[21]  Marco Cuturi,et al.  Differentiable Ranks and Sorting using Optimal Transport , 2019, 1905.11885.

[22]  Priya L. Donti,et al.  SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver , 2019, ICML.

[23]  Junchi Yan,et al.  Learning Combinatorial Embedding Networks for Deep Graph Matching , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Yoshua Bengio,et al.  Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon , 2018, Eur. J. Oper. Res..

[25]  Max Welling,et al.  Attention, Learn to Solve Routing Problems! , 2018, ICLR.

[26]  David L. Dill,et al.  Learning a SAT Solver from Single-Bit Supervision , 2018, ICLR.

[27]  Sanjeev Khanna,et al.  Better and simpler error analysis of the Sinkhorn–Knopp algorithm for matrix scaling , 2018, Mathematical Programming.

[28]  Keld Helsgaun,et al.  An Extension of the Lin-Kernighan-Helsgaun TSP Solver for Constrained Traveling Salesman and Vehicle Routing Problems: Technical report , 2017 .

[29]  Heinrich Müller,et al.  SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[31]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[32]  Jason Altschuler,et al.  Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration , 2017, NIPS.

[33]  Anoop Cherian,et al.  DeepPermNet: Visual Permutation Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  J. Zico Kolter,et al.  OptNet: Differentiable Optimization as a Layer in Neural Networks , 2017, ICML.

[35]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[36]  Sergio Verdú,et al.  Upper bounds on the relative entropy and Rényi divergence as a function of total variation distance for finite alphabets , 2015, 2015 IEEE Information Theory Workshop - Fall (ITW).

[37]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[38]  Zeyuan Allen Zhu,et al.  Using Optimization to Break the Epsilon Barrier: A Faster and Simpler Width-Independent Algorithm for Solving Positive Linear Programs in Parallel , 2014, SODA.

[39]  Brendan Pass Multi-marginal optimal transport: theory and applications , 2014, 1406.0026.

[40]  Maria Grazia Speranza,et al.  Twenty years of linear programming based portfolio optimization , 2014, Eur. J. Oper. Res..

[41]  Marco Cuturi,et al.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport , 2013, NIPS.

[42]  Ryan P. Adams,et al.  Ranking via Sinkhorn Propagation , 2011, ArXiv.

[43]  Jitendra Malik,et al.  Poselets: Body part detectors trained using 3D human pose annotations , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[44]  Baruch Awerbuch,et al.  Stateless distributed gradient descent for positive linear programs , 2008, SIAM J. Comput..

[45]  Philip A. Knight,et al.  The Sinkhorn-Knopp Algorithm: Convergence and Applications , 2008, SIAM J. Matrix Anal. Appl..

[46]  Neal E. Young,et al.  Sequential and parallel algorithms for mixed packing and covering , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[47]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[48]  W. Sharpe The Sharpe Ratio , 1994 .

[49]  Noam Nisan,et al.  A parallel approximation algorithm for positive linear programming , 1993, STOC.

[50]  David S. Johnson,et al.  Local Optimization and the Traveling Salesman Problem , 1990, ICALP.

[51]  Stephen A. Cook,et al.  The complexity of theorem-proving procedures , 1971, STOC.

[52]  Richard Sinkhorn,et al.  Concerning nonnegative matrices and doubly stochastic matrices , 1967 .

[53]  R. A. Zemlin,et al.  Integer Programming Formulation of Traveling Salesman Problems , 1960, JACM.

[54]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[55]  Hongyuan Zha,et al.  Differentiable Top-k with Optimal Transport , 2020, NeurIPS.

[56]  Francis R. Bach,et al.  Learning with Differentiable Pertubed Optimizers , 2020, NeurIPS.

[57]  W. Sharpe OF FINANCIAL AND QUANTITATIVE ANALYSIS December 1971 A LINEAR PROGRAMMING APPROXIMATION FOR THE GENERAL PORTFOLIO ANALYSIS PROBLEM , 2009 .

[58]  F. Fabozzi,et al.  Streetwise : the best of the Journal of portfolio management , 1998 .