A BLOCK COORDINATE DESCENT METHOD FOR MULTI-CONVEX OPTIMIZATION WITH APPLICATIONS TO NONNEGATIVE TENSOR FACTORIZATION AND COMPLETION

This paper considers block multi-convex optimization, where the feasible set and objective function are generally non-convex but convex in each block of variables. We review some of its interesting examples and propose a generalized block coordinate descent method. Under certain conditions, we show that any limit point satisfies the Nash equilibrium conditions. Furthermore, we establish its global convergence and estimate its asymptotic convergence rate by assuming a property based on the KurdykaLojasiewicz inequality. The proposed algorithms are adapted for factorizing nonnegative matrices and tensors, as well as completing them from their incomplete observations. The algorithms were tested on synthetic data, hyperspectral data, as well as image sets from the CBCL and ORL databases. Compared to the existing state-of-the-art algorithms, the proposed algorithms demonstrate superior performance in both speed and solution quality. The Matlab code is available for download from the authors’ homepages.

[1]  Gebräuchliche Fertigarzneimittel,et al.  V , 1893, Therapielexikon Neurologie.

[2]  Clifford Hildreth,et al.  A quadratic programming procedure , 1957 .

[3]  J. Warga.  Minimizing Certain Convex Functions , 1963 .

[4]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[5]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[6]  M. J. D. Powell,et al.  On search directions for minimization algorithms , 1973, Math. Program..

[7]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .

[8]  R. Sargent,et al.  On the convergence of sequential minimization algorithms , 1973 .

[9]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[10]  A. Auslender.  Optimisation : méthodes numériques , 1976 .

[11]  O. Mangasarian,et al.  Parallel successive overrelaxation methods for symmetric linear complementarity problems and linear programs , 1987 .

[12]  Shih-Ping Han,et al.  A successive projection method , 1988, Math. Program..

[13]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[14]  Marie-Françoise Roy,et al.  Real algebraic geometry , 1992 .

[15]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[16]  Paul Tseng,et al.  Dual coordinate ascent methods for non-strictly convex minimization , 1993, Math. Program..

[17]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[18]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[19]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[20]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[21]  Juan K. Lin,et al.  Feature extraction approach to blind source separation , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[22]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[23]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[24]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[25]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[26]  Ali Mansour,et al.  Blind Separation of Sources , 1999 .

[27]  H. Kiers.  Towards a standardized notation and terminology in multiway analysis , 2000 .

[28]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[29]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[30]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[31]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[32]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[33]  Stan Z. Li,et al.  Learning spatially localized, parts-based representation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[34]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[35]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[36]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[37]  Michael W. Berry,et al.  Text Mining Using Non-Negative Matrix Factorizations , 2004, SDM.

[38]  Seungjin Choi.  Blind Source Separation and Independent Component Analysis : A Review , 2004 .

[39]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[40]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[41]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[42]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[43]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[44]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[45]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[46]  J. Fadili,et al.  SZ and CMB reconstruction using generalized morphological component analysis , 2007, 0712.0588.

[47]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[48]  Tamara G. Kolda,et al.  Efficient MATLAB Computations with Sparse and Factored Tensors , 2007, SIAM J. Sci. Comput..

[49]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[50]  Hyunsoo Kim,et al.  Non-negative Tensor Factorization Based on Alternating Large-scale Non-negativity-constrained Least Squares , 2007, 2007 IEEE 7th International Symposium on BioInformatics and BioEngineering.

[51]  Adrian S. Lewis,et al.  Clarke Subgradients of Stratifiable Functions , 2006, SIAM J. Optim..

[52]  Seungjin Choi,et al.  Nonnegative Tucker Decomposition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[54]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[55]  Lars Kai Hansen,et al.  Algorithms for Sparse Nonnegative Tucker Decompositions , 2008, Neural Computation.

[56]  Jing Hua,et al.  Non-negative matrix factorization for semi-supervised data clustering , 2008, Knowledge and Information Systems.

[57]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..

[58]  Qiang Zhang,et al.  Tensor methods for hyperspectral data analysis: a space object material identification study. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[59]  Michael P. Friedlander,et al.  Computing non-negative tensor factorizations , 2008, Optim. Methods Softw..

[60]  Li Wang,et al.  Hybrid huberized support vector machines for microarray classification , 2007, ICML '07.

[61]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[62]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[63]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[65]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[66]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[67]  Paul Tseng,et al.  A coordinate gradient descent method for nonsmooth separable minimization , 2008, Math. Program..

[68]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[69]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[70]  Yin Zhang,et al.  An Alternating Direction Algorithm for Nonnegative Matrix Factorization , 2010 .

[71]  Yin Zhang,et al.  An alternating direction algorithm for matrix completion with nonnegative factors , 2011, Frontiers of Mathematics in China.

[72]  Ming-Jun Lai,et al.  An Unconstrained ℓq Minimization with 0 , 2011, SIAM J. Optim..

[73]  Haesun Park,et al.  Fast Nonnegative Tensor Factorization with an Active-Set-Like Method , 2012, High-Performance Scientific Computing.

[74]  Jieping Ye,et al.  Sparse non-negative tensor factorization using columnwise coordinate descent , 2012, Pattern Recognit..

[75]  M. Flatz.  Algorithms for Nonnegative Tensor Factorization , 2013 .

[76]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.