A convergence analysis of distributed dictionary learning based on the K-SVD algorithm

This paper provides a convergence analysis of a recent distributed algorithm, termed cloud K-SVD, that solves the problem of data-adaptive representations for big, distributed data. It is assumed that a number of geographically-distributed, interconnected sites have massive local data and they are collaboratively learning a sparsifying dictionary underlying these data using cloud K-SVD. This paper provides a rigorous analysis of cloud K-SVD that gives insights into its properties as well as deviations of the dictionaries learned at individual sites from a centralized solution in terms of different measures of local/global data and topology of the interconnections.

[1]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[2]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[3]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[4]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[5]  Junli Liang,et al.  Distributed Dictionary Learning for Sparse Representation in Sensor Networks , 2014, IEEE Transactions on Image Processing.

[6]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[7]  Gene H. Golub,et al.  Matrix computations , 1983 .

[8]  Cédric Richard,et al.  Learning a common dictionary over a sensor network , 2013, 2013 5th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[9]  Ali H. Sayed,et al.  Dictionary Learning Over Distributed Models , 2014, IEEE Transactions on Signal Processing.

[10]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[11]  Alexander G. Gray,et al.  Sparsity-Based Generalization Bounds for Predictive Sparse Coding , 2013, ICML.

[12]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[13]  Waheed Uz Zaman Bajwa,et al.  Cloud K-SVD: Computing data-adaptive representations in the cloud , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[14]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[15]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[16]  Waheed Uz Zaman Bajwa,et al.  Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data , 2014, IEEE Transactions on Signal Processing.