A Linearized Alternating Direction Method of Multipliers for a Special Three-Block Nonconvex Optimization Problem of Background/Foreground Extraction

In this paper, we focus on the three-block nonconvex optimization problem of background/foreground extraction from a blurred and noisy surveillance video. The coefficient matrices of the equality constraints are nonidentity matrices. Regarding the separable structure of the objective function and linear constraints, a benchmark solver for the problem is the alternating direction method of multipliers (ADMM). The computational challenge is that there is no closed-form solution to the subproblem of ADMM since the objective function is not differentiable and the coefficient matrices of the equality constraints are not identity matrices. In this paper, we propose a linearized ADMM by choosing the proximal terms appropriately and add the dual step size to make the proposed algorithm more flexible. Under proper assumptions and the associated function satisfying the Kurdyka-Łojasiewicz property, we show that the proposed algorithm converges to a critical point of the given problem. We apply the proposed algorithm to the background/foreground extraction and the numerical results are used to demonstrate the effectiveness of the proposed algorithm.

[1]  Jefferson G. Melo,et al.  Iteration-complexity of a Jacobi-type non-Euclidean ADMM for multi-block linearly constrained nonconvex programs , 2017, 1705.07229.

[2]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[3]  Qi Tian,et al.  Statistical modeling of complex backgrounds for foreground object detection , 2004, IEEE Transactions on Image Processing.

[4]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[5]  Min Li,et al.  A Symmetric Alternating Direction Method of Multipliers for Separable Nonconvex Minimization Problems , 2017, Asia Pac. J. Oper. Res..

[6]  Caihua Chen,et al.  Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights , 2015, Mathematical Programming.

[7]  Guoyin Li,et al.  Global Convergence of Splitting Methods for Nonconvex Composite Optimization , 2014, SIAM J. Optim..

[8]  Mila Nikolova,et al.  Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex Minimization , 2008, SIAM J. Imaging Sci..

[9]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[10]  Shiqian Ma,et al.  Iteration Complexity Analysis of Multi-block ADMM for a Family of Convex Minimization Without Strong Convexity , 2015, Journal of Scientific Computing.

[11]  Xiaoming Yuan,et al.  Linearized alternating direction method of multipliers for sparse group and fused LASSO models , 2014, Comput. Stat. Data Anal..

[12]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[13]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[14]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[15]  Xiaoming Yuan,et al.  On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function , 2017, Comput. Optim. Appl..

[16]  Shuicheng Yan,et al.  Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting , 2015, AAAI.

[17]  Xiaoli Yu.  Application of Linearized Alternating Direction Multiplier Method in Dictionary Learning , 2019 .

[18]  DengWei,et al.  Parallel Multi-Block ADMM with o(1 / k) Convergence , 2017 .

[19]  Wotao Yin,et al.  Global Convergence of ADMM in Nonconvex Nonsmooth Optimization , 2015, Journal of Scientific Computing.

[20]  R. Tibshirani,et al.  A note on the group lasso and a sparse group lasso , 2010, 1001.0736.

[21]  Xiaojun Chen,et al.  Alternating Direction Method of Multipliers for a Class of Nonconvex and Nonsmooth Problems with Applications to Background/Foreground Extraction , 2015, SIAM J. Imaging Sci..

[22]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[23]  Xiaoming Yuan,et al.  A Note on the Alternating Direction Method of Multipliers , 2012, J. Optim. Theory Appl..

[24]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[25]  Xiaoming Yuan,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs , 2013, SIAM J. Numer. Anal..

[26]  Tianyi Lin,et al.  On the Convergence Rate of Multi-Block ADMM , 2014, 1408.4265.

[27]  Michael C. Ferris,et al.  Operator-Splitting Methods for Monotone Affine Variational Inequalities, with a Parallel Application to Optimal Control , 1998, INFORMS J. Comput..

[28]  Radu-Emil Precup,et al.  Optimal Robot Path Planning Using Gravitational Search Algorithm , 2013 .

[29]  Mila Nikolova,et al.  Energy Minimization Methods , 2015, Handbook of Mathematical Methods in Imaging.

[30]  Deren Han,et al.  Convergence of ADMM for multi-block nonconvex separable optimization models , 2017 .

[31]  Bilal H. Abed-alguni.  Island-based Cuckoo Search with Highly Disruptive Polynomial Mutation , 2019 .

[32]  Lei Zhang,et al.  Weighted Nuclear Norm Minimization with Application to Image Denoising , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Caihua Chen,et al.  On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks , 2013 .

[34]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[35]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[36]  Kim-Chuan Toh,et al.  A Convergent 3-Block SemiProximal Alternating Direction Method of Multipliers for Conic Programming with 4-Type Constraints , 2014, SIAM J. Optim..

[37]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[38]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2014, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[39]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[40]  Xiaojun Chen,et al.  Linearly Constrained Non-Lipschitz Optimization for Image Restoration , 2015, SIAM J. Imaging Sci..

[41]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[42]  Xiaojun Chen,et al.  Smoothing Nonlinear Conjugate Gradient Method for Image Restoration Using Nonsmooth Nonconvex Minimization , 2010, SIAM J. Imaging Sci..

[43]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[44]  Deren Han,et al.  Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints , 2017, Int. J. Comput. Math..

[45]  Ali Azadeh,et al.  A hybrid meta-heuristic algorithm for optimization of crew scheduling , 2013, Appl. Soft Comput..

[46]  Deren Han,et al.  Linearized block-wise alternating direction method of multipliers for multiple-block convex programming , 2017 .

[47]  Renato D. C. Monteiro,et al.  Iteration-Complexity of Block-Decomposition Algorithms and the Alternating Direction Method of Multipliers , 2013, SIAM J. Optim..

[48]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[49]  Jinshu Su,et al.  Adaptive linearized alternating direction method of multipliers for non-convex compositely regularized optimization problems , 2017 .

[50]  Kim-Chuan Toh,et al.  A Convergent 3-Block Semi-Proximal ADMM for Convex Minimization Problems with One Strongly Convex Block , 2014, Asia Pac. J. Oper. Res..

[51]  Wotao Yin,et al.  Parallel Multi-Block ADMM with o(1 / k) Convergence , 2013, Journal of Scientific Computing.

[52]  Wei Hong Yang,et al.  Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems , 2016, SIAM J. Numer. Anal..

[53]  Thierry Bouwmans,et al.  Traditional and recent approaches in background modeling for foreground detection: An overview , 2014, Comput. Sci. Rev..

[54]  Thierry Bouwmans,et al.  Recent Advanced Statistical Background Modeling for Foreground Detection - A Systematic Survey , 2011 .

[55]  Bingsheng He,et al.  Optimally linearizing the alternating direction method of multipliers for convex programming , 2019, Comput. Optim. Appl..

[56]  Qing Ling,et al.  DLM: Decentralized Linearized Alternating Direction Method of Multipliers , 2015, IEEE Transactions on Signal Processing.