A Block Coordinate Descent Method for Regularized Multiconvex Optimization with Applications to Nonnegative Tensor Factorization and Completion

This paper considers regularized block multiconvex optimization, where the feasible set and objective function are generally nonconvex but convex in each block of variables. It also accepts nonconvex blocks and requires these blocks to be updated by proximal minimization. We review some interesting applications and propose a generalized block coordinate descent method. Under certain conditions, we show that any limit point satisfies the Nash equilibrium conditions. Furthermore, we establish global convergence and estimate the asymptotic convergence rate of the method by assuming a property based on the Kurdyka--Łojasiewicz inequality. The proposed algorithms are tested on nonnegative matrix and tensor factorization, as well as matrix and tensor recovery from incomplete observations. The tests include synthetic data and hyperspectral data, as well as image sets from the CBCL and ORL databases. Compared to the existing state-of-the-art algorithms, the proposed algorithms demonstrate superior performance in ...

[1]  J. Fadili,et al.  SZ and CMB reconstruction using generalized morphological component analysis , 2007, 0712.0588.

[2]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[3]  Hyunsoo Kim,et al.  Non-negative Tensor Factorization Based on Alternating Large-scale Non-negativity-constrained Least Squares , 2007, 2007 IEEE 7th International Symposium on BioInformatics and BioEngineering.

[4]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[5]  Qiang Zhang,et al.  Tensor methods for hyperspectral data analysis: a space object material identification study. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[6]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[7]  P. Fabry,et al.  Principal component analysis and blind source separation of modulated sources for electro-mechanical systems diagnostic , 2005 .

[8]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[9]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[10]  Chih-Jen Lin,et al.  Projected Gradient Methods for Nonnegative Matrix Factorization , 2007, Neural Computation.

[11]  R. Sargent,et al.  On the convergence of sequential minimization algorithms , 1973 .

[12]  Lars Kai Hansen,et al.  Algorithms for Sparse Nonnegative Tucker Decompositions , 2008, Neural Computation.

[13]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[14]  Ali Mansour,et al.  Blind Separation of Sources , 1999 .

[15]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[16]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[17]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[18]  Juan K. Lin,et al.  Feature extraction approach to blind source separation , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[19]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[20]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[21]  Ming-Jun Lai,et al.  An Unconstrained ℓq Minimization with 0 , 2011, SIAM J. Optim..

[22]  Adrian S. Lewis,et al.  Clarke Subgradients of Stratifiable Functions , 2006, SIAM J. Optim..

[23]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[24]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[25]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[26]  D. Donoho,et al.  Atomic Decomposition by Basis Pursuit , 2001 .

[27]  H. Kiers.  Towards a standardized notation and terminology in multiway analysis , 2000 .

[28]  Bruce L. Golden,et al.  Optimisation , 1982, IEEE Trans. Syst. Man Cybern..

[29]  Seungjin Choi,et al.  Nonnegative Tucker Decomposition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Michael W. Berry,et al.  Algorithms and applications for approximate nonnegative matrix factorization , 2007, Comput. Stat. Data Anal..

[31]  V. P. Pauca,et al.  Nonnegative matrix factorization for spectral data analysis , 2006 .

[32]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[33]  Stan Z. Li,et al.  Learning spatially localized, parts-based representation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[34]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[35]  Stefanos Zafeiriou,et al.  Algorithms for Nonnegative Tensor Factorization , 2009, Tensors in Image Processing and Computer Vision.

[36]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[37]  Jieping Ye,et al.  Sparse non-negative tensor factorization using columnwise coordinate descent , 2012, Pattern Recognit..

[38]  Michael Zibulevsky,et al.  Underdetermined blind source separation using sparse representations , 2001, Signal Process..

[39]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[40]  Paul Tseng,et al.  Dual coordinate ascent methods for non-strictly convex minimization , 1993, Math. Program..

[41]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[42]  KlingenbergBradley,et al.  Non-negative matrix factorization , 2009 .

[43]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[44]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[45]  Zhi-Quan Luo,et al.  A Unified Convergence Analysis of Block Successive Minimization Methods for Nonsmooth Optimization , 2012, SIAM J. Optim..

[46]  J. Warga.  Minimizing Certain Convex Functions , 1963 .

[47]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..

[48]  JuttenChristian,et al.  Blind separation of sources, Part 1 , 1991 .

[49]  Jing Hua,et al.  Non-negative matrix factorization for semi-supervised data clustering , 2008, Knowledge and Information Systems.

[50]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[51]  Michael P. Friedlander,et al.  Computing non-negative tensor factorizations , 2008, Optim. Methods Softw..

[52]  Tamara G. Kolda,et al.  On Tensors, Sparsity, and Nonnegative Factorizations , 2011, SIAM J. Matrix Anal. Appl..

[53]  M. J. D. Powell,et al.  On search directions for minimization algorithms , 1973, Math. Program..

[54]  Li Wang,et al.  Hybrid huberized support vector machines for microarray classification and gene selection , 2008, Bioinform..

[55]  A. Auslender.  Optimisation : méthodes numériques , 1976 .

[56]  M. Lai,et al.  An Unconstrained $\ell_q$ Minimization with $0q\leq1$ for Sparse Solution of Underdetermined Linear Systems , 2011 .

[57]  Seungjin Choi.  Blind Source Separation and Independent Component Analysis : A Review , 2004 .

[58]  Yin Zhang,et al.  An alternating direction algorithm for matrix completion with nonnegative factors , 2011, Frontiers of Mathematics in China.

[59]  O. Mangasarian,et al.  Parallel successive overrelaxation methods for symmetric linear complementarity problems and linear programs , 1987 .

[60]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[61]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[62]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[63]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[64]  Michael W. Berry,et al.  Text Mining Using Non-Negative Matrix Factorizations , 2004, SDM.

[65]  Yin Zhang,et al.  An Alternating Direction Algorithm for Nonnegative Matrix Factorization , 2010 .

[66]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[67]  Soo-Young Lee.  Blind Source Separation and Independent Component Analysis: A Review , 2005 .

[68]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[69]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[71]  Paul Tseng,et al.  A coordinate gradient descent method for nonsmooth separable minimization , 2008, Math. Program..

[72]  Marie-Françoise Roy,et al.  Real algebraic geometry , 1992 .

[73]  Andrzej Cichocki,et al.  Fast Local Algorithms for Large Scale Nonnegative Matrix and Tensor Factorizations , 2009, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[74]  Tamara G. Kolda,et al.  Efficient MATLAB Computations with Sparse and Factored Tensors , 2007, SIAM J. Sci. Comput..

[75]  Clifford Hildreth,et al.  A quadratic programming procedure , 1957 .

[76]  Haesun Park,et al.  Fast Nonnegative Tensor Factorization with an Active-Set-Like Method , 2012, High-Performance Scientific Computing.

[77]  Katya Scheinberg,et al.  Block Coordinate Descent Methods for Semidefinite Programming , 2012 .

[78]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[79]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[80]  Shih-Ping Han,et al.  A successive projection method , 1988, Math. Program..

[81]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[82]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[83]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[84]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[85]  E. Polak.  Introduction to linear and nonlinear programming , 1973 .