A New Architecture for Optimization Modeling Frameworks

We propose a new architecture for optimization modeling frameworks in which solvers are expressed as computation graphs in a framework like TensorFlow rather than as standalone programs built on a low-level linear algebra interface. Our new architecture makes it easy for modeling frameworks to support high performance computational platforms like GPUs and distributed clusters, as well as to generate solvers specialized to individual problems. Our approach is particularly well adapted to first-order and indirect optimization algorithms. We introduce cvxflow, an open-source convex optimization modeling framework in Python based on the ideas in this paper, and show that it outperforms the state of the art.

[1]  Stephen P. Boyd,et al.  A primal—dual potential reduction method for problems involving matrix inequalities , 1995, Math. Program..

[2]  Clément Farabet,et al.  Torch7: A Matlab-like Environment for Machine Learning , 2011, NIPS 2011.

[3]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[4]  Stephen P. Boyd,et al.  Convex Optimization with Abstract Linear Operators , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Jacek Gondzio,et al.  Parallel interior-point solver for structured quadratic programs: Application to financial planning problems , 2007, Ann. Oper. Res..

[6]  J. LaFountain.  Inc. , 2013, American Art.

[7]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[8]  X. Yi.  On Automatic Differentiation , 2005 .

[9]  Stephen P. Boyd,et al.  CVXPY: A Python-Embedded Modeling Language for Convex Optimization , 2016, J. Mach. Learn. Res..

[10]  Stephen P. Boyd,et al.  CVXGEN: a code generator for embedded convex optimization , 2011, Optimization and Engineering.

[11]  J. Lofberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004, 2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508).

[12]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[13]  Stephen P. Boyd,et al.  Code generation for embedded second-order cone programming , 2013, 2013 European Control Conference (ECC).

[14]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[15]  Kim-Chuan Toh,et al.  SDPT3 -- A Matlab Software Package for Semidefinite Programming , 1996 .

[16]  Stephen P. Boyd,et al.  Convex Optimization in Julia , 2014, 2014 First Workshop for High Performance Technical Computing in Dynamic Languages.

[17]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[18]  David Kendrick,et al.  GAMS, a user's guide , 1988, SGNM.

[19]  Emmanuel J. Candès,et al.  Templates for convex cone problems with applications to sparse signal recovery , 2010, Math. Program. Comput..

[20]  Frédo Durand,et al.  Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines , 2013, PLDI 2013.

[21]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[22]  J. Zico Kolter,et al.  Convex programming with fast proximal and linear operators , 2015, 1511.04815.

[23]  Christopher Fougner,et al.  Parameter Selection and Preconditioning for a Graph Form Solver , 2015, 1503.08366.

[24]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[25]  Stephen P. Boyd,et al.  ECOS: An SOCP solver for embedded systems , 2013, 2013 European Control Conference (ECC).

[26]  Griewank,et al.  On automatic differentiation , 1988 .

[27]  Brian W. Kernighan,et al.  AMPL: A Modeling Language for Mathematical Programming , 1993 .

[28]  Stephen P. Boyd,et al.  Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding , 2013, Journal of Optimization Theory and Applications.

[29]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[30]  Stephen P. Boyd,et al.  Linear Matrix Inequalities in Systems and Control Theory , 1994 .

[31]  Stephen P. Boyd,et al.  Matrix-Free Convex Optimization Modeling , 2015, 1506.00760.

[32]  Charles L. Lawson,et al.  Basic Linear Algebra Subprograms for Fortran Usage , 1979, TOMS.

[33]  A. S. Nemirovsky,et al.  Conic formulation of a convex programming problem and duality , 1992 .

[34]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[35]  Masakazu Kojima,et al.  SDPA (SemiDefinite Programming Algorithm) User's Manual Version 6.2.0 , 1995 .

[36]  Makoto Yamashita,et al.  Latest Developments in the SDPA Family for Solving Large-Scale SDPs , 2012 .

[37]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[38]  Gordon Wetzstein,et al.  ProxImaL , 2016, ACM Trans. Graph..

[39]  Stephen P. Boyd,et al.  Stochastic Matrix-Free Equilibration , 2016, J. Optim. Theory Appl..