Tensor-based algorithms for image classification

Interest in machine learning with tensor networks has been growing rapidly in recent years. We show that tensor-based methods developed for learning the governing equations of dynamical systems from data can, in the same way, be used for supervised learning problems and propose two novel approaches for image classification. One is a kernel-based reformulation of the previously introduced multidimensional approximation of nonlinear dynamics (MANDy), the other an alternating ridge regression in the tensor train format. We apply both methods to the MNIST and fashion MNIST data set and show that the approaches are competitive with state-of-the-art neural network-based classifiers.

[1]  Yinchu Zhu,et al.  Breaking the curse of dimensionality in regression , 2017, ArXiv.

[2]  Jack Hidary,et al.  TensorNetwork for Machine Learning , 2019, ArXiv.

[3]  Ivan Oseledets,et al.  A new tensor decomposition , 2009 .

[4]  David J. Schwab,et al.  Supervised Learning with Tensor Networks , 2016, NIPS.

[5]  David J. Schwab,et al.  Supervised Learning with Quantum-Inspired Tensor Networks , 2016, ArXiv.

[6]  Stefan Klus,et al.  Multidimensional Approximation of Nonlinear Dynamical Systems , 2018, Journal of Computational and Nonlinear Dynamics.

[7]  C. W. Groetsch,et al.  Inverse Problems in the Mathematical Sciences , 1993 .

[8]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[9]  K. Birgitta Whaley,et al.  Towards quantum machine learning with tensor networks , 2018, Quantum Science and Technology.

[10]  Maria Schuld,et al.  Quantum Machine Learning in Feature Hilbert Spaces. , 2018, Physical review letters.

[11]  Markus Weimar.  Breaking the curse of dimensionality , 2015 .

[12]  A. Zhdanov.  The method of augmented regularized normal equations , 2012 .

[13]  Gene H. Golub,et al.  Matrix computations , 1983 .

[14]  HansenPer Christian.  The truncated SVD as a method for regularization , 1987 .

[15]  Stefan Klus,et al.  Nearest-neighbor interaction systems in the tensor-train format , 2016, J. Comput. Phys..

[16]  E. Miles Stoudenmire,et al.  Learning relevant features of data with multi-scale tensor networks , 2017, ArXiv.

[17]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[18]  Xiao Zhang,et al.  Entanglement-Based Feature Extraction by Tensor Network Machine Learning , 2018, Frontiers in Applied Mathematics and Statistics.

[19]  Christof Schütte,et al.  Solving the master equation without kinetic Monte Carlo: Tensor train approximations for a CO oxidation model , 2016, J. Comput. Phys..

[20]  Stefan Klus,et al.  Tensor-based dynamic mode decomposition , 2016, Nonlinearity.

[21]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[22]  Adam Zalcman,et al.  TensorNetwork: A Library for Physics and Machine Learning , 2019, ArXiv.

[23]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[24]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[25]  White,et al.  Density matrix formulation for quantum renormalization groups. , 1992, Physical review letters.

[26]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[27]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[28]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[29]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[30]  Feliks Nüske,et al.  Tensor-based EDMD for the Koopman analysis of high-dimensional systems , 2019, ArXiv.

[31]  Steven L. Brunton,et al.  Data-driven discovery of partial differential equations , 2016, Science Advances.

[32]  J. C. A. Barata,et al.  The Moore–Penrose Pseudoinverse: A Tutorial Review of the Theory , 2011, 1110.6882.

[33]  Martin J. Mohlenkamp,et al.  Multivariate Regression and Machine Learning with Sums of Separable Functions , 2009, SIAM J. Sci. Comput..