Fast Primal-Dual Gradient Method for Strongly Convex Minimization Problems with Linear Constraints

In this paper we consider a class of optimization problems with a strongly convex objective function and the feasible set given by an intersection of a simple convex set with a set given by a number of linear equality and inequality constraints. A number of optimization problems in applications can be stated in this form, examples being the entropy-linear programming, the ridge regression, the elastic net, the regularized optimal transport, etc. We extend the Fast Gradient Method applied to the dual problem in order to make it primal-dual so that it allows not only to solve the dual problem, but also to construct nearly optimal and nearly feasible solution of the primal problem. We also prove a theorem about the convergence rate for the proposed algorithm in terms of the objective function and the linear constraints infeasibility.

[1]  S. Fang,et al.  Entropy Optimization and Mathematical Programming , 1997 .

[2]  Marco Cuturi,et al.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport , 2013, NIPS.

[3]  Gabriel Peyré,et al.  Iterative Bregman Projections for Regularized Transportation Problems , 2014, SIAM J. Sci. Comput..

[4]  Y. Nesterov,et al.  Efficient numerical methods for entropy-linear programming problems , 2016, Computational Mathematics and Mathematical Physics.

[5]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[6]  Attahiru Sule Alfa,et al.  IP Traffic Matrix Estimation Methods: Comparisons and Improvements , 2006, 2006 IEEE International Conference on Communications.

[7]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[8]  L. Bregman The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[9]  Roman A. Polyak,et al.  Dual fast projected gradient method for quadratic programming , 2013, Optim. Lett..

[10]  J. Lorenz,et al.  On the scaling of multidimensional matrices , 1989 .

[11]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[12]  J. N. Kapur Maximum-entropy models in science and engineering , 1992 .

[13]  Johan A. K. Suykens,et al.  Application of a Smoothing Technique to Decomposition in Convex Optimization , 2008, IEEE Transactions on Automatic Control.

[14]  Carsten Lund,et al.  Estimating point-to-point and point-to-multipoint traffic matrices: an information-theoretic approach , 2005, IEEE/ACM Transactions on Networking.

[15]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[16]  L. Bregman Proof of the convergence of Sheleikhovskii's method for a problem with transportation constraints , 1967 .

[17]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[18]  Alexander Gasnikov,et al.  Entropy linear programming , 2014 .

[19]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[20]  Yurii Nesterov,et al.  First-order methods of smooth convex optimization with inexact oracle , 2013, Mathematical Programming.

[21]  Douglas J. Miller,et al.  Maximum entropy econometrics: robust estimation with limited data , 1996 .