Matrix-Free Convex Optimization Modeling

We introduce a convex optimization modeling framework that transforms a convex optimization problem expressed in a form natural and convenient for the user into an equivalent cone program in a way that preserves fast linear transforms in the original problem. By representing linear functions in the transformation process not as matrices, but as graphs that encode composition of linear operators, we arrive at a matrix-free cone program, i.e., one whose data matrix is represented by a linear operator and its adjoint. This cone program can then be solved by a matrix-free cone solver. By combining the matrix-free modeling framework and cone solver, we obtain a general method for efficiently solving convex optimization problems involving fast linear transforms.

[1]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[2]  Paul Dienes,et al.  On H-matrices , 1952 .

[3]  G. Rota,et al.  A note on the joint spectral radius , 1960 .

[4]  J. Tukey,et al.  An algorithm for the machine calculation of complex Fourier series , 1965 .

[5]  Peter D. Welch,et al.  The Fast Fourier Transform and Its Applications , 1969 .

[6]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[7]  N. Ahmed,et al.  Discrete Cosine Transform , 1996 .

[8]  P. Krishnaprasad,et al.  A descent approach to a class of inverse problems , 1977 .

[9]  Eljas SOISALON-SOININEN On the Covering Problem for Left-Recursive Grammars , 1979, Theor. Comput. Sci..

[10]  Daniel Brélaz,et al.  New methods to color the vertices of a graph , 1979, CACM.

[11]  R. Bracewell The fast Hartley transform , 1984, Proceedings of the IEEE.

[12]  A.K. Krishnamurthy,et al.  Multidimensional digital signal processing , 1985, Proceedings of the IEEE.

[13]  Wolfgang Hackbusch,et al.  Multi-grid methods and applications , 1985, Springer series in computational mathematics.

[14]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[15]  Leslie Greengard,et al.  A fast algorithm for particle simulations , 1987 .

[16]  I. Duff,et al.  Direct Methods for Sparse Matrices , 1987 .

[17]  I. Daubechies Orthonormal bases of compactly supported wavelets , 1988 .

[18]  Griewank,et al.  On automatic differentiation , 1988 .

[19]  Stephen P. Boyd,et al.  Structured and Simultaneous Lyapunov Functions for System Stability Problems , 1989 .

[20]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Jorge Herbert de Lira,et al.  Two-Dimensional Signal and Image Processing , 1989 .

[22]  Leslie Greengard,et al.  The Fast Gauss Transform , 1991, SIAM J. Sci. Comput..

[23]  Daubechies,et al.  Ten Lectures on Wavelets Volume 921 , 1992 .

[24]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[25]  C. Loan Computational Frameworks for the Fast Fourier Transform , 1992 .

[26]  A. S. Nemirovsky,et al.  Conic formulation of a convex programming problem and duality , 1992 .

[27]  Alan J. Laub,et al.  Solution of the Sylvester matrix equation AXBT + CXDT = E , 1992, TOMS.

[28]  Jelena Kovacevic,et al.  Nonseparable multidimensional perfect reconstruction filter banks and wavelet bases for Rn , 1992, IEEE Trans. Inf. Theory.

[29]  I. Daubechies,et al.  Biorthogonal bases of compactly supported wavelets , 1992 .

[30]  Magnús M. Hallórsson A still better performance guarantee for approximate graph coloring , 1993 .

[31]  M. Halldórsson A Still Better Performance Guarantee for Approximate Graph Coloring , 1993, Inf. Process. Lett..

[32]  Stephen P. Boyd,et al.  A Polynomial-time Algorithm for Determining Quadratic Lyapunov Functions for Nonlinear Systems , 1993 .

[33]  Stephen P. Boyd,et al.  Linear Matrix Inequalities in Systems and Control Theory , 1994 .

[34]  Stephen A. Martucci,et al.  Symmetric convolution and the discrete sine and cosine transforms , 1993, IEEE Trans. Signal Process..

[35]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[36]  Stephen P. Boyd,et al.  A primal—dual potential reduction method for problems involving matrix inequalities , 1995, Math. Program..

[37]  Masakazu Kojima,et al.  SDPA (SemiDefinite Programming Algorithm) User's Manual Version 6.2.0 , 1995 .

[38]  Kim-Chuan Toh,et al.  SDPT3 -- A Matlab Software Package for Semidefinite Programming , 1996 .

[39]  Stephen J. Wright Primal-Dual Interior-Point Methods , 1997, Other Titles in Applied Mathematics.

[40]  E. Yaz Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[41]  Yinyu Ye,et al.  Interior point algorithms: theory and analysis , 1997 .

[42]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[43]  W. Hackbusch A Sparse Matrix Arithmetic Based on $\Cal H$-Matrices. Part I: Introduction to ${\Cal H}$-Matrices , 1999, Computing.

[44]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[45]  Wolfgang Hackbusch,et al.  A Sparse Matrix Arithmetic Based on H-Matrices. Part I: Introduction to H-Matrices , 1999, Computing.

[46]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[47]  W. Hackbusch,et al.  On H2-Matrices , 2000 .

[48]  Yinyu Ye,et al.  Solving Sparse Semidefinite Programs Using the Dual Scaling Algorithm with an Iterative Solver , 2000 .

[49]  Emmanuel J. Candès,et al.  The curvelet transform for image denoising , 2002, IEEE Trans. Image Process..

[50]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[51]  Jos F. Sturm,et al.  Implementation of interior point methods for mixed semidefinite and second order cone optimization problems , 2002, Optim. Methods Softw..

[52]  Masakazu Kojima,et al.  Lagrangian Dual Interior-Point Methods for Semidefinite Programs , 2002, SIAM J. Optim..

[53]  Minh N. Do,et al.  The finite ridgelet transform for image representation , 2003, IEEE Trans. Image Process..

[54]  W. Hackbusch,et al.  Introduction to Hierarchical Matrices with Applications , 2003 .

[55]  Kim-Chuan Toh,et al.  Solving semidefinite-quadratic-linear programs using SDPT3 , 2003, Math. Program..

[56]  Anders la Cour-Harbo,et al.  Ripples in Mathematics , 2003 .

[57]  Larry S. Davis,et al.  Improved fast gauss transform and efficient kernel density estimation , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[58]  Larry S. Davis,et al.  Efficient Kernel Machines Using the Improved Fast Gauss Transform , 2004, NIPS.

[59]  Shang-Hua Teng,et al.  Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems , 2003, STOC '04.

[60]  Daniel D. Lee,et al.  Nonnegative deconvolution for time of arrival estimation , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[61]  M. Kojima,et al.  SDPA-C (SemiDefinite Programming Algorithm - Completion method) User's Manual — Version 6.2.0 , 2004 .

[62]  J. Lofberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004, 2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508).

[63]  Kim-Chuan Toh,et al.  Solving Large Scale Semidefinite Programs via an Iterative Solver on the Augmented Systems , 2003, SIAM J. Optim..

[64]  Lexing Ying,et al.  3D discrete curvelet transform , 2005, SPIE Optics + Photonics.

[65]  Yinyu Ye,et al.  DSDP5: Software for Semidefinite Programming , 2005 .

[66]  X. Yi On Automatic Differentiation , 2005 .

[67]  Y. Nesterov,et al.  On the accuracy of the ellipsoid norm approximation of the joint spectral radius , 2005 .

[68]  Timothy A. Davis,et al.  Direct Methods for Sparse Linear Systems (Fundamentals of Algorithms 2) , 2006 .

[69]  Laurent Demanet,et al.  Fast Discrete Curvelet Transforms , 2006, Multiscale Model. Simul..

[70]  N. Maculan,et al.  Global optimization : from theory to implementation , 2006 .

[71]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[72]  Mila Nikolova,et al.  Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models , 2006, SIAM J. Appl. Math..

[73]  Y. Nesterov Towards Nonsymmetric Conic Optimization , 2006 .

[74]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools (2nd Edition) , 2006 .

[75]  Stephen P. Boyd,et al.  Disciplined Convex Programming , 2006 .

[76]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[77]  Jacek Gondzio,et al.  Parallel interior-point solver for structured quadratic programs: Application to financial planning problems , 2007, Ann. Oper. Res..

[78]  Minh N. Do,et al.  Multidimensional Directional Filter Banks and Surfacelets , 2007, IEEE Transactions on Image Processing.

[79]  R. Vershynin,et al.  One sketch for all: fast algorithms for compressed sensing , 2007, STOC '07.

[80]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[81]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[82]  KoăźVaraMichal,et al.  On the solution of large-scale SDP problems by the modified barrier method using iterative solvers , 2007 .

[83]  Yinyu Ye,et al.  Algorithm 875: DSDP5—software for semidefinite programming , 2008, TOMS.

[84]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[85]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[86]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[87]  Michael Stingl,et al.  On the solution of large-scale SDP problems by the modified barrier method using iterative solvers , 2009, Math. Program..

[88]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[89]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[90]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[91]  Michael A. Saunders,et al.  LSMR: An Iterative Algorithm for Sparse Least-Squares Problems , 2010, SIAM J. Sci. Comput..

[92]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[93]  Kim-Chuan Toh,et al.  A Newton-CG Augmented Lagrangian Method for Semidefinite Programming , 2010, SIAM J. Optim..

[94]  Renato D. C. Monteiro,et al.  Primal-dual first-order methods with $${\mathcal {O}(1/\epsilon)}$$ iteration-complexity for cone programming , 2011, Math. Program..

[95]  Michael A. Saunders,et al.  LSMR: An Iterative Algorithm for Sparse Least-Squares Problems , 2011, SIAM J. Sci. Comput..

[96]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[97]  Guanghui Lan,et al.  Primal-dual first-order methods with O (1/e) iteration-complexity for cone programming. , 2011 .

[98]  Emmanuel J. Candès,et al.  Templates for convex cone problems with applications to sparse signal recovery , 2010, Math. Program. Comput..

[99]  Clément Farabet,et al.  Torch7: A Matlab-like Environment for Machine Learning , 2011, NIPS 2011.

[100]  Laurent Jacques,et al.  A panorama on multiscale geometric representations, intertwining spatial, directional and frequency selectivity , 2011, Signal Process..

[101]  Jianqin Zhou,et al.  On discrete cosine transform , 2011, ArXiv.

[102]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[103]  Zhang Liu,et al.  Interior-point methods for large-scale cone programming , 2011 .

[104]  Yurii Nesterov,et al.  Towards non-symmetric conic optimization , 2012, Optim. Methods Softw..

[105]  Stephen P. Boyd,et al.  CVXGEN: a code generator for embedded convex optimization , 2011, Optimization and Engineering.

[106]  Jacek Gondzio,et al.  Matrix-free interior point method , 2012, Comput. Optim. Appl..

[107]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[108]  Zeyuan Allen Zhu,et al.  A simple, combinatorial algorithm for solving SDD systems in nearly-linear time , 2013, STOC '13.

[109]  Edo Liberty,et al.  Simple and deterministic matrix sketching , 2012, KDD.

[110]  Stephen P. Boyd,et al.  ECOS: An SOCP solver for embedded systems , 2013, 2013 European Control Conference (ECC).

[111]  Jacek Gondzio,et al.  Convergence Analysis of an Inexact Feasible Interior Point Method for Convex Quadratic Programming , 2012, SIAM J. Optim..

[112]  Stephen P. Boyd,et al.  A Primal-Dual Operator Splitting Method for Conic Optimization , 2013 .

[113]  Frédo Durand,et al.  Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines , 2013, PLDI 2013.

[114]  Stephen P. Boyd,et al.  Code generation for embedded second-order cone programming , 2013, 2013 European Control Conference (ECC).

[115]  Stephen P. Boyd,et al.  Convex Optimization in Julia , 2014, 2014 First Workshop for High Performance Technical Computing in Dynamic Languages.

[116]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[117]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[118]  Jacek Gondzio,et al.  Matrix-free interior point method for compressed sensing problems , 2012, Mathematical Programming Computation.

[119]  Yinyu Ye,et al.  A homogeneous interior-point algorithm for nonsymmetric convex conic optimization , 2014, Mathematical Programming.

[120]  Stephen P. Boyd,et al.  Convex Optimization with Abstract Linear Operators , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[121]  Le Thi Khanh Hien,et al.  Differential properties of Euclidean projection onto power cone , 2015, Math. Methods Oper. Res..

[122]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[123]  Stephen P. Boyd,et al.  Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding , 2013, Journal of Optimization Theory and Applications.

[124]  Stephen P. Boyd,et al.  CVXPY: A Python-Embedded Modeling Language for Convex Optimization , 2016, J. Mach. Learn. Res..

[125]  Stephen P. Boyd,et al.  Stochastic Matrix-Free Equilibration , 2016, J. Optim. Theory Appl..

[126]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[127]  Christopher Fougner,et al.  Parameter Selection and Preconditioning for a Graph Form Solver , 2015, 1503.08366.

[128]  J. CARRIERt,et al.  A FAST ADAPTIVE MULTIPOLE ALGORITHM FOR PARTICLE SIMULATIONS * , 2022 .