Rate-improved inexact augmented Lagrangian method for constrained nonconvex optimization

First-order methods have been studied for nonlinear constrained optimization within the framework of the augmented Lagrangian method (ALM) or penalty method. We propose an improved inexact ALM (iALM) and conduct a unified analysis for nonconvex problems with either convex or nonconvex constraints. Under certain regularity conditions (that are also assumed by existing works), we show an $\tilde{O}(\varepsilon^{-\frac{5}{2}})$ complexity result for a problem with a nonconvex objective and convex constraints and an $\tilde{O}(\varepsilon^{-3})$ complexity result for a problem with a nonconvex objective and nonconvex constraints, where the complexity is measured by the number of first-order oracles to yield an $\varepsilon$-KKT solution. Both results are the best known. The same-order complexity results have been achieved by penalty methods. However, two different analysis techniques are used to obtain the results, and more importantly, the penalty methods generally perform significantly worse than iALM in practice. Our improved iALM and analysis close the gap between theory and practice. Numerical experiments are provided to demonstrate the effectiveness of our proposed method.

[1]  L. Hien,et al.  An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems , 2017, J. Optim. Theory Appl..

[2]  Saeed Ghadimi,et al.  Accelerated gradient methods for nonconvex nonlinear and stochastic programming , 2013, Mathematical Programming.

[3]  Yangyang Xu,et al.  Inexact Proximal-Point Penalty Methods for Constrained Non-Convex Optimization , 2019 .

[4]  Renato D. C. Monteiro,et al.  Iteration-complexity of first-order augmented Lagrangian methods for convex programming , 2015, Mathematical Programming.

[5]  Ion Necoara,et al.  Computational Complexity of Inexact Gradient Augmented Lagrangian Methods: Application to Constrained MPC , 2013, SIAM J. Control. Optim..

[6]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[7]  Weiwei Kong,et al.  Complexity of a Quadratic Penalty Accelerated Inexact Proximal Point Method for Solving Linearly Constrained Nonconvex Composite Programs , 2018, SIAM J. Optim..

[8]  Hao Yu,et al.  A Simple Parallel Algorithm with an O(1/t) Convergence Rate for General Convex Programs , 2015, SIAM J. Optim..

[9]  Chih-Jen Lin,et al.  Iteration complexity of feasible descent methods for convex optimization , 2014, J. Mach. Learn. Res..

[10]  Yangyang Xu,et al.  Primal-Dual Stochastic Gradient Method for Convex Programs with Many Functional Constraints , 2018, SIAM J. Optim..

[11]  Yangyang Xu,et al.  Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming , 2017, Mathematical Programming.

[12]  Erfan Yazdandoost Hamedani,et al.  A Primal-Dual Algorithm with Line Search for General Convex-Concave Saddle Point Problems , 2020, SIAM J. Optim..

[13]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[14]  Zheng Qu,et al.  An inexact proximal augmented Lagrangian framework with arbitrary linearly convergent inner solver for composite convex optimization , 2019, Mathematical Programming Computation.

[15]  Yunmei Chen,et al.  An Accelerated Linearized Alternating Direction Method of Multipliers , 2014, SIAM J. Imaging Sci..

[16]  Le Thi Khanh Hien,et al.  A primal-dual smoothing gap reduction framework for strongly convex-generally concave saddle point problems , 2017 .

[17]  Rahul Singh,et al.  Alternating Gradient Descent Ascent for Nonconvex Min-Max Problems in Robust Learning and GANs , 2019, 2019 53rd Asilomar Conference on Signals, Systems, and Computers.

[18]  Shiqian Ma,et al.  Structured nonconvex and nonsmooth optimization: algorithms and iteration complexity analysis , 2016, Computational Optimization and Applications.

[19]  N. S. Aybat,et al.  A Primal-Dual Algorithm for General Convex-Concave Saddle Point Problems , 2018, 1803.01401.

[20]  N. S. Aybat,et al.  An Augmented Lagrangian Method for Conic Convex Programming , 2013, 1302.6322.

[21]  Michael I. Jordan,et al.  Near-Optimal Algorithms for Minimax Optimization , 2020, COLT.

[22]  Yongxin Chen,et al.  Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applications , 2019, IEEE Transactions on Signal Processing.

[23]  Jefferson G. Melo,et al.  Iteration-complexity of an inexact proximal accelerated augmented Lagrangian method for solving linearly constrained smooth nonconvex composite optimization problems , 2020, 2006.08048.

[24]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[25]  Hao Yu,et al.  A primal-dual type algorithm with the O(1/t) convergence rate for large scale constrained convex programs , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[26]  Yangyang Xu,et al.  First-order methods for constrained convex programming based on linearized augmented Lagrangian function , 2017, INFORMS J. Optim..

[27]  Naoki Katoh,et al.  Resource Allocation Problems , 1998 .

[28]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[29]  Volkan Cevher,et al.  An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraints , 2019, NeurIPS.

[30]  Qihang Lin,et al.  A Level-Set Method for Convex Optimization with a Feasible Solution Path , 2018, SIAM J. Optim..

[31]  Nicholas I. M. Gould,et al.  On the complexity of finding first-order critical points in constrained nonlinear optimization , 2014, Math. Program..

[32]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[33]  Bo Yang,et al.  SNAP: Finding Approximate Second-Order Stationary Solutions Efficiently for Non-convex Linearly Constrained Problems , 2019, ArXiv.

[34]  Yair Carmon,et al.  Accelerated Methods for NonConvex Optimization , 2018, SIAM J. Optim..

[35]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[36]  Zhaosong Lu,et al.  Iteration-complexity of first-order augmented Lagrangian methods for convex conic programming , 2018, SIAM J. Optim..

[37]  Songtao Lu,et al.  Finding Second-Order Stationary Points Efficiently in Smooth Nonconvex Linearly Constrained Optimization Problems , 2020, NeurIPS.

[38]  Jiawei Zhang,et al.  A Global Dual Error Bound and Its Application to the Analysis of Linearly Constrained Nonconvex Optimization , 2020, SIAM J. Optim..

[39]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Composite Function Minimization with Applications to Nonconvex Nonlinear Programming , 2011, SIAM J. Optim..

[40]  Ion Necoara,et al.  Rate Analysis of Inexact Dual First-Order Methods Application to Dual Decomposition , 2014, IEEE Transactions on Automatic Control.

[41]  Yangyang Xu,et al.  Augmented Lagrangian based first-order methods for convex and nonconvex programs: nonergodic convergence and iteration complexity , 2020, ArXiv.