Tensor Decompositions in Deep Learning

The paper surveys the topic of tensor decompositions in modern machine learning applications. It focuses on three active research topics of significant relevance for the community. After a brief review of consolidated works on multi-way data analysis, we consider the use of tensor decompositions in compressing the parameter space of deep learning models. Lastly, we discuss how tensor methods can be leveraged to yield richer adaptive representations of complex data, including structured information. The paper concludes with a discussion on interesting open research challenges.

[1]  Amnon Shashua,et al.  Convolutional Rectifier Networks as Generalized Tensor Decompositions , 2016, ICML.

[2]  Davide Bacciu,et al.  An input-output hidden Markov model for tree transductions , 2013, Neurocomputing.

[3]  Davide Bacciu,et al.  Generalising Recursive Neural Models by Tensor Decomposition , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[4]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[5]  Davide Bacciu,et al.  A Gentle Introduction to Deep Learning for Graphs , 2019, Neural Networks.

[6]  J. Chen,et al.  Equivalence of restricted Boltzmann machines and tensor network states , 2017, 1701.04831.

[7]  Davide Bacciu,et al.  A Fair Comparison of Graph Neural Networks for Graph Classification , 2020, ICLR.

[8]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[9]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[10]  Hans-Peter Kriegel,et al.  Factorizing YAGO: scalable machine learning for linked data , 2012, WWW.

[11]  H. Kiers.  Towards a standardized notation and terminology in multiway analysis , 2000 .

[12]  Niranjan Balasubramanian,et al.  Event Representations with Tensor-based Compositions , 2017, AAAI.

[13]  Zenglin Xu,et al.  TensorD: A tensor decomposition library in TensorFlow , 2018, Neurocomputing.

[14]  Amnon Shashua,et al.  Inductive Bias of Deep Convolutional Networks through Pooling Geometry , 2016, ICLR.

[15]  Vijaya Saradhi Vedula,et al.  Mining Temporal Changes in Strengths and Weaknesses of Cricket Players Using Tensor Decomposition , 2020, ESANN.

[16]  Konstantinos Kalpakis,et al.  Inferring relations in knowledge graphs with tensor decompositions , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[17]  Davide Bacciu,et al.  Compositional Generative Mapping for Tree-Structured Data—Part I: Bottom-Up Probabilistic Modeling of Trees , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Hans-Peter Kriegel,et al.  A Three-Way Model for Collective Learning on Multi-Relational Data , 2011, ICML.

[19]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[20]  Masashi Sugiyama,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives , 2017, Found. Trends Mach. Learn..

[21]  Christos Faloutsos,et al.  Spotting misbehaviors in location-based social networks using tensors , 2014, WWW.

[22]  Stefano Soatto,et al.  Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks , 2017, 2018 Information Theory and Applications Workshop (ITA).

[23]  Xuan Li,et al.  A Survey on Tensor Techniques and Applications in Machine Learning , 2019, IEEE Access.

[24]  Danilo P. Mandic,et al.  Tucker Tensor Layer in Fully Connected Neural Networks , 2019, ArXiv.

[25]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[26]  Maja Pantic,et al.  TensorLy: Tensor Learning in Python , 2016, J. Mach. Learn. Res..

[27]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[28]  Christopher Potts,et al.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[29]  Matthieu Cord,et al.  MUTAN: Multimodal Tucker Fusion for Visual Question Answering , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[31]  Ivan V. Oseledets,et al.  Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition , 2014, ICLR.

[32]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[33]  Davide Bacciu,et al.  Tensor Decompositions in Recursive Neural Networks for Tree-Structured Data , 2020, ESANN.

[34]  Yongxin Yang,et al.  Deep Multi-task Representation Learning: A Tensor Factorisation Approach , 2016, ICLR.

[35]  Davide Bacciu,et al.  Theoretically Expressive and Edge-aware Graph Learning , 2020, ESANN.

[36]  Davide Bacciu,et al.  Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing , 2018, ICML.

[37]  CichockiAndrzej,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization , 2016 .

[38]  Davide Bacciu,et al.  Compositional Generative Mapping for Tree-Structured Data—Part II: Topographic Projection Model , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Davide Bacciu,et al.  Bayesian Tensor Factorisation for Bottom-up Hidden Tree Markov Models , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).