Alternating proximal gradient method for sparse nonnegative Tucker decomposition

Multi-way data arises in many applications such as electroencephalography classification, face recognition, text mining and hyperspectral data analysis. Tensor decomposition has been commonly used to find the hidden factors and elicit the intrinsic structures of the multi-way data. This paper considers sparse nonnegative Tucker decomposition (NTD), which is to decompose a given tensor into the product of a core tensor and several factor matrices with sparsity and nonnegativity constraints. An alternating proximal gradient method is applied to solve the problem. The algorithm is then modified to sparse NTD with missing values. Per-iteration cost of the algorithm is estimated scalable about the data size, and global convergence is established under fairly loose conditions. Numerical experiments on both synthetic and real world data demonstrate its superiority over a few state-of-the-art methods for (sparse) NTD from partial and/or full observations. The MATLAB code along with demos are accessible from the author’s homepage.

[1]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[2]  Stefanos Zafeiriou,et al.  Discriminant Nonnegative Tensor Factorization Algorithms , 2009, IEEE Transactions on Neural Networks.

[3]  Yin Zhang,et al.  An Alternating Direction Algorithm for Nonnegative Matrix Factorization , 2010 .

[4]  Jieping Ye,et al.  Sparse non-negative tensor factorization using columnwise coordinate descent , 2012, Pattern Recognit..

[5]  Seungjin Choi,et al.  Nonnegative Tucker Decomposition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Andrzej Cichocki,et al.  Nonnegative Tensor Factorization for Continuous EEG Classification , 2007, Int. J. Neural Syst..

[7]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[8]  L. K. Hansen,et al.  Algorithms for Sparse Non-negative Tucker decompositions , 2008 .

[9]  Hyunsoo Kim,et al.  Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method , 2008, SIAM J. Matrix Anal. Appl..

[10]  H. Kiers.  Joint Orthomax Rotation of the Core and Component Matrices Resulting from Three-mode Principal Components Analysis , 1998 .

[11]  Andrzej Cichocki,et al.  Extended HALS algorithm for nonnegative Tucker decomposition and its applications for multiway analysis and classification , 2011, Neurocomputing.

[12]  Tze Meng Low,et al.  Exploiting Symmetry in Tensors for High Performance , 2013, ArXiv.

[13]  Marie-Françoise Roy,et al.  Real algebraic geometry , 1992 .

[14]  Tapani Ristaniemi,et al.  Feature Extraction by Nonnegative Tucker Decomposition from EEG Data Including Testing and Training Observations , 2012, ICONIP.

[15]  Michael P. Friedlander,et al.  Computing non-negative tensor factorizations , 2008, Optim. Methods Softw..

[16]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[17]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[18]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[19]  Guillermo Sapiro,et al.  Classification and clustering via dictionary learning with structured incoherence and shared features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Yin Zhang,et al.  An alternating direction algorithm for matrix completion with nonnegative factors , 2011, Frontiers of Mathematics in China.

[21]  Qing Ling,et al.  Decentralized low-rank matrix completion , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[23]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[24]  Tamara G. Kolda,et al.  MATLAB Tensor Toolbox , 2006 .

[25]  Xu Yangyang.  A BLOCK COORDINATE DESCENT METHOD FOR MULTI-CONVEX OPTIMIZATION WITH APPLICATIONS TO NONNEGATIVE TENSOR FACTORIZATION AND COMPLETION , 2012 .

[26]  Max Welling,et al.  Positive tensor factorization , 2001, Pattern Recognit. Lett..

[27]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[28]  Qiang Zhang,et al.  Tensor methods for hyperspectral data analysis: a space object material identification study. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[29]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[30]  Lars Kai Hansen,et al.  Algorithms for Sparse Nonnegative Tucker Decompositions , 2008, Neural Computation.

[31]  A. Cichocki,et al.  Damped Gauss-Newton algorithm for nonnegative Tucker decomposition , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[32]  Hyunsoo Kim,et al.  Non-negative Tensor Factorization Based on Alternating Large-scale Non-negativity-constrained Least Squares , 2007, 2007 IEEE 7th International Symposium on BioInformatics and BioEngineering.

[33]  Springer-Verlag London Limited.  Temporal relation co-clustering on directional social network and author-topic evolution , 2010 .

[34]  Tze Meng Low,et al.  Exploiting Symmetry in Tensors for High Performance: Multiplication with Symmetric Tensors , 2013, SIAM J. Sci. Comput..

[35]  Adrian S. Lewis,et al.  THEINEQUALITY FOR NONSMOOTH SUBANALYTIC FUNCTIONS WITH APPLICATIONS TO , 2007 .

[36]  Victoria Stodden,et al.  When Does Non-Negative Matrix Factorization Give a Correct Decomposition into Parts? , 2003, NIPS.

[37]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[38]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[39]  P. Tseng.  Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization , 2001 .

[40]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[41]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[43]  Haesun Park,et al.  Toward Faster Nonnegative Matrix Factorization: A New Algorithm and Comparisons , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[44]  Stefanos Zafeiriou,et al.  Algorithms for Nonnegative Tensor Factorization , 2009, Tensors in Image Processing and Computer Vision.

[45]  R. Tyrrell Rockafellar,et al.  Variational Analysis , 1998, Grundlehren der mathematischen Wissenschaften.

[46]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[47]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[48]  Genevera I. Allen,et al.  Sparse Higher-Order Principal Components Analysis , 2012, AISTATS.