An inertial Bregman generalized alternating direction method of multipliers for nonconvex optimization

In this paper, a class of nonconvex optimization with linearly constrained is considered. An inertial Bregman generalized alternating direction method of multiplies is investigated for solving the nonconvex optimization. The iterative schemes are formulated in the spirit of the proximal alternating direction method of multipliers and its inertial variant. The proximal term is introduced via Bregman distance, a fact that allows us to derive new proximal splitting algorithms for large-scale separable optimization problems. Under some assumptions, we prove that the iterative sequence generated by the algorithm converges to a critical point of the considered problem. Finally, we report some preliminary numerical results on solving signal recovery and SCAD penalty problems to verify the efficiency of the proposed method.

[1]  Miantao Chao,et al.  Convergence of the Peaceman-Rachford Splitting Method for a Class of Nonconvex Programs , 2021, Numerical Mathematics: Theory, Methods and Applications.

[2]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[3]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[4]  Min Li,et al.  A Symmetric Alternating Direction Method of Multipliers for Separable Nonconvex Minimization Problems , 2017, Asia Pac. J. Oper. Res..

[5]  Thomas Pock,et al.  Inertial Proximal Alternating Linearized Minimization (iPALM) for Nonconvex and Nonsmooth Problems , 2016, SIAM J. Imaging Sci..

[6]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[7]  Jefferson G. Melo,et al.  Iteration-complexity analysis of a generalized alternating direction method of multipliers , 2017, J. Glob. Optim..

[8]  Deren Han,et al.  Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints , 2017, Int. J. Comput. Math..

[9]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[10]  Jefferson G. Melo,et al.  Convergence rate bounds for a proximal ADMM with over-relaxation stepsize parameter for solving nonconvex linearly constrained problems , 2017, 1702.01850.

[11]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[12]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[13]  Shiqian Ma,et al.  Inertial Proximal ADMM for Linearly Constrained Separable Convex Optimization , 2015, SIAM J. Imaging Sci..

[14]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[15]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[16]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[17]  D. R. Sahu,et al.  Inertial iterative algorithms for common solution of variational inequality and system of variational inequalities problems , 2020, Journal of Applied Mathematics and Computing.

[18]  Nicolas Gillis,et al.  Inertial Block Proximal Methods for Non-Convex Non-Smooth Optimization , 2019, ICML.

[19]  Chongshou Li,et al.  Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems , 2020, Journal of Global Optimization.

[20]  Zongben Xu,et al.  $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast Solver , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Ling Li,et al.  Alternating direction method of multipliers for nonconvex fused regression problems , 2019, Comput. Stat. Data Anal..

[22]  Guoyin Li,et al.  Global Convergence of Splitting Methods for Nonconvex Composite Optimization , 2014, SIAM J. Optim..

[23]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[24]  Hong-Kun Xu,et al.  Convergence of Bregman alternating direction method with multipliers for nonconvex composite problems , 2014, 1410.8625.

[25]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[26]  Xingju Cai,et al.  A Gauss–Seidel type inertial proximal alternating linearized minimization for a class of nonconvex optimization problems , 2020, J. Glob. Optim..

[27]  Huan Gao,et al.  A hybrid Bregman alternating direction method of multipliers for the linearly constrained difference-of-convex problems , 2020, J. Glob. Optim..

[28]  S. K. Zavriev,et al.  Heavy-ball method in nonconvex optimization problems , 1993 .

[29]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[30]  Radu Ioan Bot,et al.  An inertial forward–backward algorithm for the minimization of the sum of two nonconvex functions , 2014, EURO J. Comput. Optim..

[31]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[32]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[33]  Deren Han,et al.  O(1/t) complexity analysis of the generalized alternating direction method of multipliers , 2019 .

[34]  Jieping Ye,et al.  Large-scale sparse logistic regression , 2009, KDD.

[35]  M. T. Chao,et al.  An inertial proximal alternating direction method of multipliers for nonconvex optimization , 2020, Int. J. Comput. Math..

[36]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[37]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[38]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[39]  Radu Ioan Bot,et al.  The Proximal Alternating Direction Method of Multipliers in the Nonconvex Setting: Convergence Analysis and Rates , 2018, Math. Oper. Res..

[40]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[41]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[42]  Bingsheng He,et al.  Generalized alternating direction method of multipliers: new theoretical insights and applications , 2015, Math. Program. Comput..

[43]  Caihua Chen,et al.  A General Inertial Proximal Point Algorithm for Mixed Variational Inequality Problem , 2015, SIAM J. Optim..

[44]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .