Efficient dictionary learning via very sparse random projections

Performing signal processing tasks on compressive measurements of data has received great attention in recent years. In this paper, we extend previous work on compressive dictionary learning by showing that more general random projections may be used, including sparse ones. More precisely, we examine compressive K-means clustering as a special case of compressive dictionary learning and give theoretical guarantees for its performance for a very general class of random projections. We then propose a memory and computation efficient dictionary learning algorithm, specifically designed for analyzing large volumes of high-dimensional data, which learns the dictionary from very sparse random projections. Experimental results demonstrate that our approach allows for reduction of computational complexity and memory/data access, with controllable loss in accuracy.

[1]  Yonina C. Eldar,et al.  Blind Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[2]  Volkan Cevher,et al.  Low-Dimensional Models for Dimensionality Reduction and Signal Recovery: A Geometric Perspective , 2010, Proceedings of the IEEE.

[3]  Michael B. Wakin,et al.  Sketched SVD: Recovering Spectral Features from Compressive Measurements , 2012, ArXiv.

[4]  Lei Zhang,et al.  Fast Compressive Tracking , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Guillermo Sapiro,et al.  Supervised Dictionary Learning , 2008, NIPS.

[6]  Urbashi Mitra,et al.  2008 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2008 .

[7]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[8]  Michael Elad,et al.  Efficient Implementation of the K-SVD Algorithm using Batch Orthogonal Matching Pursuit , 2008 .

[9]  Michael Elad,et al.  On the Role of Sparse and Redundant Representations in Image Processing , 2010, Proceedings of the IEEE.

[10]  E.J. Candes,et al.  An Introduction To Compressive Sampling , 2008, IEEE Signal Processing Magazine.

[11]  James E. Fowler,et al.  Compressive-Projection Principal Component Analysis , 2009, IEEE Transactions on Image Processing.

[12]  Shannon M. Hughes,et al.  Efficient recovery of principal components from compressive measurements with application to Gaussian mixture model estimation , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Trac D. Tran,et al.  Fast and Efficient Compressive Sensing Using Structurally Random Matrices , 2011, IEEE Transactions on Signal Processing.

[14]  Waheed Uz Zaman Bajwa,et al.  Cloud K-SVD: Computing data-adaptive representations in the cloud , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[15]  Richard G. Baraniuk,et al.  Signal Processing With Compressive Measurements , 2010, IEEE Journal of Selected Topics in Signal Processing.

[16]  Yonina C. Eldar,et al.  Blind Compressed Sensing Over a Structured Union of Subspaces , 2011, ArXiv.

[17]  Stephen J. Wright,et al.  Computational Methods for Sparse Solution of Linear Inverse Problems , 2010, Proceedings of the IEEE.

[18]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[19]  Shannon M. Hughes,et al.  Compressive K-SVD , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[21]  George Atia,et al.  Change Detection with Compressive Measurements , 2014, IEEE Signal Processing Letters.

[22]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[23]  Shannon M. Hughes,et al.  Memory and Computation Efficient PCA via Very Sparse Random Projections , 2014, ICML.

[24]  Hanchao Qi,et al.  Invariance of principal components under low-dimensional random projection of the data , 2012, 2012 19th IEEE International Conference on Image Processing.

[25]  Jean Ponce,et al.  Task-Driven Dictionary Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Kjersti Engan,et al.  Recursive Least Squares Dictionary Learning Algorithm , 2010, IEEE Transactions on Signal Processing.

[27]  Richard G. Baraniuk,et al.  Dictionary learning from sparsely corrupted or compressed signals , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).