A general system for heuristic minimization of convex functions over non-convex sets

We describe general heuristics to approximately solve a wide variety of problems with convex objective and decision variables from a non-convex set. The heuristics, which employ convex relaxations, convex restrictions, local neighbour search methods, and the alternating direction method of multipliers, require the solution of a modest number of convex problems, and are meant to apply to general problems, without much tuning. We describe an implementation of these methods in a package called NCVX, as an extension of CVXPY, a Python package for formulating and solving convex optimization problems. We study several examples of well known non-convex problems, and show that our general purpose heuristics are effective in finding approximate solutions to a wide variety of problems.

[1]  W. Sharpe Portfolio Theory and Capital Markets , 1970 .

[2]  Nate Derbinsky,et al.  An Improved Three-Weight Message-Passing Algorithm , 2013, ArXiv.

[3]  Nikolaos V. Sahinidis,et al.  A polyhedral branch-and-cut approach to global optimization , 2005, Math. Program..

[4]  Stephen P. Boyd,et al.  Linear models based on noisy data and the Frisch scheme , 2013, SIAM Rev..

[5]  Stephen P. Boyd,et al.  Variations and extension of the convex–concave procedure , 2016 .

[6]  Hector J. Levesque,et al.  Hard and Easy Distributions of SAT Problems , 1992, AAAI.

[7]  João M. F. Xavier,et al.  Basis Pursuit in sensor networks , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[9]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[10]  Mhand Hifi,et al.  A Literature Review on Circle and Sphere Packing Problems: Models and Methodologies , 2009, Adv. Oper. Res..

[11]  Giovanni Rinaldi,et al.  A Branch-and-Cut Algorithm for the Resolution of Large-Scale Symmetric Traveling Salesman Problems , 1991, SIAM Rev..

[12]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[13]  Stephen P. Boyd,et al.  CVXPY: A Python-Embedded Modeling Language for Convex Optimization , 2016, J. Mach. Learn. Res..

[14]  Warren P. Adams,et al.  A hierarchy of relaxation between the continuous and convex hull representations , 1990 .

[15]  Jianli Chen,et al.  A proximal alternating direction method of multipliers for a minimization problem with nonconvex constraints , 2015, J. Glob. Optim..

[16]  M. Hestenes Multiplier and gradient methods , 1969 .

[17]  JeongKyu Lee A Graph-Based Approach for Modeling and Indexing Video Data , 2006, Eighth IEEE International Symposium on Multimedia (ISM'06).

[18]  Stephen P. Boyd,et al.  A simple effective heuristic for embedded mixed-integer quadratic programming , 2015, 2016 American Control Conference (ACC).

[19]  James M. Crawford,et al.  Experimental Results on the Crossover Point in Random 3-SAT , 1996, Artif. Intell..

[20]  Edwin R. Hancock,et al.  Inexact graph matching using genetic search , 1997, Pattern Recognit..

[21]  Nikos D. Sidiropoulos,et al.  Consensus-ADMM for General Quadratically Constrained Quadratic Programming , 2016, IEEE Transactions on Signal Processing.

[22]  Jon W. Tolle,et al.  Exact penalty functions in nonlinear programming , 1973, Math. Program..

[23]  Pablo A. Parrilo,et al.  Diagonal and Low-Rank Matrix Decompositions, Correlation Matrices, and Ellipsoid Fitting , 2012, SIAM J. Matrix Anal. Appl..

[24]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[25]  Philip N. Klein,et al.  Recognition of shapes by editing their shock graphs , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Mario Vento,et al.  Thirty Years Of Graph Matching In Pattern Recognition , 2004, Int. J. Pattern Recognit. Artif. Intell..

[27]  Gregory Gutin,et al.  The traveling salesman problem , 2006, Discret. Optim..

[28]  Hanif D. Sherali,et al.  A Hierarchy of Relaxations Between the Continuous and Convex Hull Representations for Zero-One Programming Problems , 1990, SIAM J. Discret. Math..

[29]  James W. Cannon,et al.  Introduction to circle packing: the theory of discrete analytic functions , 2007 .

[30]  Nikos D. Sidiropoulos,et al.  Parallel Algorithms for Constrained Tensor Factorization via Alternating Direction Method of Multipliers , 2014, IEEE Transactions on Signal Processing.

[31]  Y. Aflalo,et al.  On convex relaxation of graph isomorphism , 2015, Proceedings of the National Academy of Sciences.

[32]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[33]  Stephen P. Boyd,et al.  Diagonal scaling in Douglas-Rachford splitting and ADMM , 2014, 53rd IEEE Conference on Decision and Control.

[34]  Stephen P. Boyd,et al.  Preconditioning in fast dual gradient methods , 2014, 53rd IEEE Conference on Decision and Control.

[35]  Shinji Umeyama,et al.  An Eigendecomposition Approach to Weighted Graph Matching Problems , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[37]  Guoyin Li,et al.  Global Convergence of Splitting Methods for Nonconvex Composite Optimization , 2014, SIAM J. Optim..

[38]  Alejandro Ribeiro,et al.  Consensus in Ad Hoc WSNs With Noisy Links—Part I: Distributed Estimation of Deterministic Signals , 2008, IEEE Transactions on Signal Processing.

[39]  Mingyi Hong,et al.  A Distributed, Asynchronous, and Incremental Algorithm for Nonconvex Optimization: An ADMM Approach , 2014, IEEE Transactions on Control of Network Systems.

[40]  Christoph Schnörr,et al.  Evaluation of Convex Optimization Techniques for the Weighted Graph-Matching Problem in Computer Vision , 2001, DAGM-Symposium.

[41]  Brendt Wohlberg,et al.  A nonconvex ADMM algorithm for group sparsity with sparse groups , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[42]  Stephen P. Boyd,et al.  MIMO PID tuning via iterated LMI restriction , 2016 .

[43]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[44]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[45]  Stephen P. Boyd,et al.  Monotonicity and restart in fast gradient methods , 2014, 53rd IEEE Conference on Decision and Control.

[46]  Theodosios Pavlidis,et al.  A shape analysis model with applications to a character recognition system , 1992, [1992] Proceedings IEEE Workshop on Applications of Computer Vision.

[47]  Rick Chartrand,et al.  Nonconvex Splitting for Regularized Low-Rank + Sparse Decomposition , 2012, IEEE Transactions on Signal Processing.

[48]  J. Kruskal On the shortest spanning subtree of a graph and the traveling salesman problem , 1956 .

[49]  N. Biggs THE TRAVELING SALESMAN PROBLEM A Guided Tour of Combinatorial Optimization , 1986 .

[50]  Keinosuke Fukunaga,et al.  A Branch and Bound Algorithm for Feature Subset Selection , 1977, IEEE Transactions on Computers.

[51]  Tomaso Erseghe,et al.  Distributed Optimal Power Flow Using ADMM , 2014, IEEE Transactions on Power Systems.

[52]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[53]  János D. Pintér,et al.  Solving circle packing problems by global optimization: Numerical results and industrial applications , 2008, Eur. J. Oper. Res..

[54]  Kenneth Stephenson,et al.  A circle packing algorithm , 2003, Comput. Geom..

[55]  R. Merris Laplacian matrices of graphs: a survey , 1994 .

[56]  William J. Cook,et al.  Solution of a Large-Scale Traveling-Salesman Problem , 1954, 50 Years of Integer Programming.

[57]  E. L. Lawler,et al.  Branch-and-Bound Methods: A Survey , 1966, Oper. Res..

[58]  William G. Wee,et al.  Object Recognition and Recovery by Skeleton Graph Matching , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[59]  Shiqian Ma,et al.  Alternating direction method of multipliers for real and complex polynomial optimization models , 2014 .

[60]  Anima Anandkumar,et al.  Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition , 2014, NIPS.

[61]  Mario Vento,et al.  A large database of graphs and its use for benchmarking graph isomorphism algorithms , 2003, Pattern Recognit. Lett..

[62]  R. Kálmán Identification of noisy systems , 1985 .

[63]  Jonathan E. Spingarn,et al.  Applications of the method of partial inverses to convex programming: Decomposition , 1985, Math. Program..

[64]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[65]  T. D. Morley,et al.  Eigenvalues of the Laplacian of a graph , 1985 .

[66]  M. J. D. Powell,et al.  Algorithms for nonlinear constraints that use lagrangian functions , 1978, Math. Program..

[67]  Roger Fletcher,et al.  An exact penalty function for nonlinear programming with inequalities , 1973, Math. Program..

[68]  Sanjay Mehrotra,et al.  A branch-and-cut method for 0-1 mixed convex programming , 1999, Math. Program..

[69]  Carey E. Priebe,et al.  Fast Approximate Quadratic Programming for Graph Matching , 2015, PloS one.

[70]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[71]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[72]  Huchuan Lu,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON IMAGE PROCESSING 1 Online Object Tracking with Sparse Prototypes , 2022 .

[73]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[74]  Michael Goldberg The Packing of Equal Circles in a Square , 1970 .

[75]  Anton van den Hengel,et al.  Semidefinite Programming , 2014, Computer Vision, A Reference Guide.

[76]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[77]  B. De Moor,et al.  The opposite of analytic centering for solving minimum rank problems in control and identification , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[78]  Yin Zhang,et al.  An alternating direction algorithm for matrix completion with nonnegative factors , 2011, Frontiers of Mathematics in China.

[79]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[80]  Jonathan Eckstein,et al.  Understanding the Convergence of the Alternating Direction Method of Multipliers: Theoretical and Computational Perspectives , 2015 .

[81]  Hatem Hmam,et al.  Quadratic Optimisation with One Quadratic Equality Constraint , 2010 .

[82]  Peter Brucker,et al.  A Branch and Bound Algorithm for the Job-Shop Scheduling Problem , 1994, Discret. Appl. Math..

[83]  Necdet Serhat Aybat,et al.  An ADMM Algorithm for Clustering Partially Observed Networks , 2014, SDM.

[84]  TWO-WEEK Loan COpy,et al.  University of California , 1886, The American journal of dental science.

[85]  L. Grippo,et al.  Exact penalty functions in constrained optimization , 1989 .

[86]  André F. Perold,et al.  Large-Scale Portfolio Optimization , 1984 .

[87]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[88]  Hong-Kun Xu,et al.  Convergence of Bregman alternating direction method with multipliers for nonconvex composite problems , 2014, 1410.8625.