Online Mixed-Integer Optimization in Milliseconds

We propose a method to solve online mixed-integer optimization (MIO) problems at very high speed using machine learning. By exploiting the repetitive nature of online optimization, we are able to greatly speedup the solution time. Our approach encodes the optimal solution into a small amount of information denoted as strategy using the Voice of Optimization framework proposed in [BS18]. In this way the core part of the optimization algorithm becomes a multiclass classification problem which can be solved very quickly. In this work we extend that framework to real-time and high-speed applications focusing on parametric mixed-integer quadratic optimization (MIQO). We propose an extremely fast online optimization algorithm consisting of a feedforward neural network (NN) evaluation and a linear system solution where the matrix has already been factorized. Therefore, this online approach does not require any solver nor iterative algorithm. We show the speed of the proposed method both in terms of total computations required and measured execution time. We estimate the number of floating point operations (flops) required to completely recover the optimal solution as a function of the problem dimensions. Compared to state-of-the-art MIO routines, the online running time of our method is very predictable and can be lower than a single matrix factorization time. We benchmark our method against the state-of-the-art solver Gurobi obtaining from two to three orders of magnitude speedups on benchmarks with real-world data.

[1]  Stephen P. Boyd,et al.  Multi-Period Trading via Convex Optimization , 2017, Found. Trends Optim..

[2]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[3]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[4]  Stephen P. Boyd,et al.  ECOS: An SOCP solver for embedded systems , 2013, 2013 European Control Conference (ECC).

[5]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[6]  Dimitris Bertsimas,et al.  The voice of optimization , 2018, Machine Learning.

[7]  Stephen P. Boyd,et al.  OSQP: an operator splitting solver for quadratic programs , 2017, 2018 UKACC 12th International Conference on Control (CONTROL).

[8]  Stephen P. Boyd,et al.  CVXPY: A Python-Embedded Modeling Language for Convex Optimization , 2016, J. Mach. Learn. Res..

[9]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[10]  Stephen P. Boyd,et al.  CVXGEN: a code generator for embedded convex optimization , 2011, Optimization and Engineering.

[11]  Line Roald,et al.  Learning for Constrained Optimization: Identifying Optimal Active Constraint Sets , 2018, INFORMS J. Comput..

[12]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[13]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[14]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[15]  Le Song,et al.  Discriminative Embeddings of Latent Variable Models for Structured Data , 2016, ICML.

[16]  John N. Tsitsiklis,et al.  Introduction to linear optimization , 1997, Athena scientific optimization and computation series.

[17]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[18]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[19]  Stephen P. Boyd,et al.  A general system for heuristic minimization of convex functions over non-convex sets , 2017, Optim. Methods Softw..

[20]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[21]  Florian Herzog,et al.  Stochastic Model Predictive Control And Portfolio Optimization , 2007 .

[22]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.

[23]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[24]  Stephen P. Boyd,et al.  Embedded code generation using the OSQP solver , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[25]  Joao P. S. Catalao,et al.  Scheduling of head-dependent cascaded hydro systems: mixed-integer quadratic programming approach , 2010 .

[26]  Alberto Bemporad,et al.  The explicit linear quadratic regulator for constrained systems , 2003, Autom..

[27]  Dimitris Bertsimas,et al.  Robust multiperiod portfolio management in the presence of transaction costs , 2008, Comput. Oper. Res..

[28]  Alberto Bemporad,et al.  Control of systems integrating logic, dynamics, and constraints , 1999, Autom..

[29]  Manfred Morari,et al.  Embedded optimization for mixed logical dynamical systems , 2015, Comput. Chem. Eng..

[30]  Dimitris Bertsimas,et al.  Optimization over integers , 2005 .

[31]  I. Good THE POPULATION FREQUENCIES OF SPECIES AND THE ESTIMATION OF POPULATION PARAMETERS , 1953 .

[32]  Andrea Lodi,et al.  Learning a Classification of Mixed-Integer Quadratic Programming Problems , 2017, CPAIOR.

[33]  Le Song,et al.  2 Common Formulation for Greedy Algorithms on Graphs , 2018 .

[34]  Le Song,et al.  Learning to Branch in Mixed Integer Programming , 2016, AAAI.

[35]  Jeremy Kepner,et al.  Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[36]  Michal Kvasnica,et al.  Machine learning-based warm starting of active set methods in embedded model predictive control , 2019, Eng. Appl. Artif. Intell..

[37]  Christian Kirches,et al.  qpOASES: a parametric active-set algorithm for quadratic programming , 2014, Mathematical Programming Computation.

[38]  Mohamed Oussama Damen,et al.  Lattice code decoder for space-time codes , 2000, IEEE Communications Letters.

[39]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[40]  Fred W. Glover,et al.  The feasibility pump , 2005, Math. Program..

[41]  Yoshua Bengio,et al.  Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon , 2018, Eur. J. Oper. Res..

[42]  Robert E. Bixby,et al.  A Brief History of Linear and Mixed-Integer Programming Computation , 2012 .