Tensor Train Neighborhood Preserving Embedding

In this paper, we propose a tensor train neighborhood preserving embedding (TTNPE) to embed multidimensional tensor data into low-dimensional tensor subspace. Novel approaches to solve the optimization problem in TTNPE are proposed. For this embedding, we evaluate a novel tradeoff gain among classification, computation, and dimensionality reduction (storage) for supervised learning. It is shown that compared to the state-of-the-arts tensor embedding methods, TTNPE achieves superior tradeoff in classification, computation, and dimensionality reduction in MNIST handwritten digits, Weizmann face datasets, and financial market datasets.

[1]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[2]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[3]  Minh N. Do,et al.  Efficient Tensor Completion for Color Image and Video Recovery: Low-Rank Tensor Train , 2016, IEEE Transactions on Image Processing.

[4]  Jiasong Wu,et al.  Multilinear Principal Component Analysis Network for Tensor Object Classification , 2014, IEEE Access.

[5]  N. Altman.  An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression , 1992 .

[6]  Minh N. Do,et al.  Efficient tensor completion: Low-rank tensor train , 2016, ArXiv.

[7]  Shuicheng Yan,et al.  Neighborhood preserving embedding , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[8]  Amir Beck,et al.  On the Convergence of Alternating Minimization for Convex Programming with Applications to Iteratively Reweighted Least Squares and Decomposition Schemes , 2015, SIAM J. Optim..

[9]  Alexander Novikov,et al.  Ultimate tensorization: compressing convolutional and FC layers alike , 2016, ArXiv.

[10]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[11]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[12]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[13]  Yousef Saad,et al.  Fast methods for estimating the Numerical rank of large matrices , 2016, ICML.

[14]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[15]  Deng Cai,et al.  Tensor Subspace Analysis , 2005, NIPS.

[16]  M. Moslehian.  Ky Fan inequalities , 2011, 1108.1467.

[17]  Liqing Zhang,et al.  Tensor Ring Decomposition , 2016, ArXiv.

[18]  Satoshi Nakamura,et al.  Compressing recurrent neural network with tensor train , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[19]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[20]  Haiping Lu,et al.  Multilinear Principal Component Analysis of Tensor Objects for Recognition , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[21]  Dit-Yan Yeung,et al.  Tensor Embedding Methods , 2006, AAAI.

[22]  Demetri Terzopoulos,et al.  Multilinear subspace analysis of image ensembles , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[23]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[24]  V. Aggarwal,et al.  Efficient Low Rank Tensor Ring Completion , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Vaneet Aggarwal,et al.  Tensor Completion by Alternating Minimization under the Tensor Train (TT) Model , 2016, ArXiv.

[26]  Xiaodong Wang,et al.  Deterministic and Probabilistic Conditions for Finite Completability of Low-Tucker-Rank Tensor , 2016, IEEE Transactions on Information Theory.

[27]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[28]  Yifan Sun,et al.  Wide Compression: Tensor Ring Nets , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[30]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[31]  Prabin Kumar Bora,et al.  An Efficient SVD Shrinkage for Rank Estimation , 2015, IEEE Signal Processing Letters.

[32]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..