Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization

We study the convergence properties of a (block) coordinate descent method applied to minimize a nondifferentiable (nonconvex) function f(x1, . . . , xN) with certain separability and regularity properties. Assuming that f is continuous on a compact level set, the subsequence convergence of the iterates to a stationary point is shown when either f is pseudoconvex in every pair of coordinate blocks from among N-1 coordinate blocks or f has at most one minimum in each of N-2 coordinate blocks. If f is quasiconvex and hemivariate in every coordinate block, then the assumptions of continuity of f and compactness of the level set may be relaxed further. These results are applied to derive new (and old) convergence results for the proximal minimization algorithm, an algorithm of Arimoto and Blahut, and an algorithm of Han. They are applied also to a problem of blind source separation.

[1]  Clifford Hildreth,et al.  A quadratic programming procedure , 1957 .

[2]  J. Warga.  Minimizing Certain Convex Functions , 1963 .

[3]  L. Bregman.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming , 1967 .

[4]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[5]  Norman Zadeh.  Note---A Note on the Cyclic Coordinate Ascent Method , 1970 .

[6]  R. E. Larson,et al.  A dynamic programming successive approximations technique with convergence proofs , 1970 .

[7]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[8]  Suguru Arimoto,et al.  An algorithm for computing the capacity of arbitrary discrete memoryless channels , 1972, IEEE Trans. Inf. Theory.

[9]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[10]  Richard E. Blahut,et al.  Computation of channel capacity and rate-distortion functions , 1972, IEEE Trans. Inf. Theory.

[11]  M. J. D. Powell,et al.  On search directions for minimization algorithms , 1973, Math. Program..

[12]  R. Glowinski,et al.  Sur des méthodes d'optimisation par relaxation , 1973 .

[13]  R. Sargent,et al.  On the convergence of sequential minimization algorithms , 1973 .

[14]  N. G. F. Sancho,et al.  A new algorithm for the solution of multi-state dynamic programming problems , 1975, Math. Program..

[15]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[16]  A. Auslender.  Optimisation : méthodes numériques , 1976 .

[17]  T. Stern,et al.  A Class of Decentralized Routing Algorithms Using Relaxation , 1977, IEEE Trans. Commun..

[18]  O. Mangasarian,et al.  Parallel successive overrelaxation methods for symmetric linear complementarity problems and linear programs , 1987 .

[19]  Z. Zuo,et al.  Successive approximation technique for a class of large-scale NLP problems and its application to dynamic programming , 1989 .

[20]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[21]  Shih-Ping Han,et al.  A Decomposition Method and Its Application to Convex Programming , 1989, Math. Oper. Res..

[22]  P. Tseng.  Dual ascent methods for problems with strictly convex costs and linear constraints: a unified approach , 1990 .

[23]  Paul Tseng,et al.  Dual coordinate ascent methods for non-strictly convex minimization , 1993, Math. Program..

[24]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[25]  Paul Tseng,et al.  On the Convergence Rate of Dual Ascent Methods for Linearly Constrained Convex Minimization , 1993, Math. Oper. Res..

[26]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[27]  Krzysztof C. Kiwiel,et al.  Free-Steering Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints , 1997, Math. Oper. Res..

[28]  Barak A. Pearlmutter,et al.  Blind source separation by sparse decomposition , 2000, SPIE Defense + Commercial Sensing.

[29]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[30]  Andrzej Stachurski,et al.  Parallel Optimization: Theory, Algorithms and Applications , 2000, Parallel Distributed Comput. Pract..

[31]  P. Tseng,et al.  Block Coordinate Relaxation Methods for Nonparametric Wavelet Denoising , 2000 .