Sample complexity bounds for dictionary learning of tensor data

This paper provides bounds on the sample complexity of estimating Kronecker-structured dictionaries for Kth-order tensor data. The training samples are generated by linear combinations of these structured dictionary atoms and observed through white Gaussian noise. The lower bound follows from a lower bound on the minimax risk for general coefficient distributions and can be further specialized to sparse-Gaussian coefficients. This bound scales linearly with the sum of the product of the dimensions of the (smaller) coordinate dictionaries for tensor data. An explicit dictionary estimation algorithm for 2nd-order tensor data is also provided whose sample complexity matches the lower bound in the scaling sense. Numerical experiments highlight the advantages associated with explicitly accounting for tensor structure of data during dictionary learning.

[1]  Prateek Jain,et al.  Learning Sparsely Used Overcomplete Dictionaries , 2014, COLT.

[2]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[3]  Sanjeev Arora,et al.  New Algorithms for Learning Incoherent and Overcomplete Dictionaries , 2013, COLT.

[4]  Anand D. Sarwate,et al.  Minimax lower bounds for Kronecker-structured dictionary learning , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[5]  Anand D. Sarwate,et al.  Minimax Lower Bounds on Dictionary Learning for Tensor Data , 2016, IEEE Transactions on Information Theory.

[6]  Florian Roemer,et al.  Tensor-based algorithms for learning multidimensional separable dictionaries , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[7]  Martin Kleinsteuber,et al.  Separable Dictionary Learning , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Yi Yang,et al.  Decomposable Nonlocal Tensor Dictionary Learning for Multispectral Image Denoising , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Rajat Raina,et al.  Self-taught learning: transfer learning from unlabeled data , 2007, ICML '07.

[10]  Bin Yu.  Assouad, Fano, and Le Cam , 1997 .

[11]  Bennett B. Borden,et al.  Finding the Signal in the Noise: Information Governance, Analytics, and the Future of Legal Practice , 2014 .

[12]  AgarwalAlekh,et al.  A Clustering Approach to Learning Sparsely Used Overcomplete Dictionaries , 2017 .

[13]  Karin Schnass,et al.  On the Identifiability of Overcomplete Dictionaries via the Minimisation Principle Underlying K-SVD , 2013, ArXiv.

[14]  Yen-Wei Chen,et al.  K-CPD: Learning of overcomplete dictionaries for tensor sparse coding , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[15]  Yonina C. Eldar,et al.  On the Minimax Risk of Dictionary Learning , 2015, IEEE Transactions on Information Theory.

[16]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[17]  Yonina C. Eldar,et al.  Performance limits of dictionary learning for sparse coding , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[18]  Misha Elena Kilmer,et al.  A tensor-based dictionary learning approach to tomographic image reconstruction , 2015, BIT Numerical Mathematics.

[19]  Jean Ponce,et al.  Task-Driven Dictionary Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Karin Schnass,et al.  Local identification of overcomplete dictionaries , 2014, J. Mach. Learn. Res..

[21]  Rémi Gribonval,et al.  Sparse and Spurious: Dictionary Learning With Noise and Outliers , 2014, IEEE Transactions on Information Theory.

[22]  A. Bruckstein,et al.  On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them , 2006 .

[23]  Syed Zubair,et al.  Tensor dictionary learning with sparse TUCKER decomposition , 2013, 2013 18th International Conference on Digital Signal Processing (DSP).

[24]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[25]  Zemin Zhang,et al.  Denoising and Completion of 3D Data via Multidimensional Dictionary Learning , 2015, IJCAI.