High-order tensor completion via gradient-based optimization under tensor train format

Abstract Tensor train (TT) decomposition has drawn people’s attention due to its powerful representation ability and performance stability in high-order tensors. In this paper, we propose a novel approach to recover the missing entries of incomplete data represented by higher-order tensors. We attempt to find the low-rank TT decomposition of the incomplete data which captures the latent features of the whole data and then reconstruct the missing entries. By applying gradient descent algorithms, tensor completion problem is efficiently solved by optimization models. We propose two TT-based algorithms: Tensor Train Weighted Optimization (TT-WOPT) and Tensor Train Stochastic Gradient Descent (TT-SGD) to optimize TT decomposition factors. In addition, a method named Visual Data Tensorization (VDT) is proposed to transform visual data into higher-order tensors, resulting in the performance improvement of our algorithms. The experiments in synthetic data and visual data show high efficiency and performance of our algorithms compared to the state-of-the-art completion algorithms, especially in high-order, high missing rate, and large-scale tensor completion situations.

[1]  V. Aggarwal,et al.  Efficient Low Rank Tensor Ring Completion , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Amnon Shashua,et al.  Linear image coding for regression and classification using the tensor-rank principle , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[3]  B. Khoromskij.  Tensor numerical methods for multidimensional PDES: theoretical analysis and initial applications , 2015 .

[4]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Steffen Staab,et al.  TripleRank: Ranking Semantic Web Data by Tensor Decomposition , 2009, SEMWEB.

[6]  Lieven De Lathauwer,et al.  Blind Identification of Underdetermined Mixtures by Simultaneous Matrix Diagonalization , 2008, IEEE Transactions on Signal Processing.

[7]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[8]  Ce Zhu,et al.  Image Completion Using Low Tensor Tree Rank and Total Variation Minimization , 2019, IEEE Transactions on Multimedia.

[9]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[10]  Anima Anandkumar,et al.  Online and Differentially-Private Tensor Decomposition , 2016, NIPS.

[11]  Salah Bourennane,et al.  Multidimensional filtering based on a tensor approach , 2005, Signal Process..

[12]  Tamara G. Kolda,et al.  Poblano v1.0: A Matlab Toolbox for Gradient-Based Optimization , 2010 .

[13]  L. Lathauwer,et al.  On the Best Rank-1 and Rank-( , 2004 .

[14]  David D. Cox,et al.  Tensor Switching Networks , 2016, NIPS.

[15]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[16]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[17]  Lieven De Lathauwer,et al.  Optimization-Based Algorithms for Tensor Decompositions: Canonical Polyadic Decomposition, Decomposition in Rank-(Lr, Lr, 1) Terms, and a New Generalization , 2013, SIAM J. Optim..

[18]  Demetri Terzopoulos,et al.  Multilinear subspace analysis of image ensembles , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[20]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[21]  J. Mocks,et al.  Topographic components model for event-related potentials and some biophysical considerations , 1988, IEEE Transactions on Biomedical Engineering.

[22]  Peter J. Haas,et al.  Large-scale matrix factorization with distributed stochastic gradient descent , 2011, KDD.

[23]  Tamara G. Kolda,et al.  Scalable Tensor Factorizations for Incomplete Data , 2010, ArXiv.

[24]  Marko Filipovic,et al.  Tucker factorization with missing data with application to low-$$n$$n-rank tensor completion , 2015, Multidimens. Syst. Signal Process..

[25]  Volker Tresp,et al.  Tensor-Train Recurrent Neural Networks for Video Classification , 2017, ICML.

[26]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[27]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[28]  Pierre Comon,et al.  Tensor CP Decomposition With Structured Factor Matrices: Algorithms and Performance , 2016, IEEE Journal of Selected Topics in Signal Processing.

[29]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[30]  Liqing Zhang,et al.  Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Minh N. Do,et al.  Efficient Tensor Completion for Color Image and Video Recovery: Low-Rank Tensor Train , 2016, IEEE Transactions on Image Processing.

[32]  Jianting Cao,et al.  Completion of High Order Tensor Data with Missing Entries via Tensor-Train Decomposition , 2017, ICONIP.

[33]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[34]  Liangpei Zhang,et al.  Hyperspectral Image Restoration Using Low-Rank Matrix Recovery , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[35]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[36]  Ken-ichi Kawarabayashi,et al.  Expected Tensor Decomposition with Stochastic Gradient Descent , 2016, AAAI.