Greedy Deep Dictionary Learning

In this work we propose a new deep learning tool called deep dictionary learning. Multi-level dictionaries are learnt in a greedy fashion, one layer at a time. This requires solving a simple (shallow) dictionary learning problem, the solution to this is well known. We apply the proposed technique on some benchmark deep learning datasets. We compare our results with other deep learning tools like stacked autoencoder and deep belief network; and state of the art supervised dictionary learning tools like discriminative KSVD and label consistent KSVD. Our method yields better results than all.

[1]  Svetha Venkatesh,et al.  Joint learning and dictionary construction for pattern recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Michael Elad,et al.  Image Sequence Denoising via Sparse and Redundant Representations , 2009, IEEE Transactions on Image Processing.

[3]  Rabab K. Ward,et al.  Multiresolution Methods in Face Recognition , 2008 .

[4]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[5]  Trac D. Tran,et al.  Hyperspectral Image Classification via Kernel Sparse Representation , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Shuzhi Sam Ge,et al.  Analysis of Different Sparsity Methods in Constrained RBM for Sparse Representation in Cognitive Robotic Perception , 2015, Journal of Intelligent & Robotic Systems.

[7]  Rabab Kreidieh Ward,et al.  Improved Group Sparse Classifier , 2010, Pattern Recognit. Lett..

[8]  Michael Elad,et al.  Image Denoising Via Learned Dictionaries and Sparse representation , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[9]  Rabab Kreidieh Ward,et al.  Learning space-time dictionaries for blind compressed sensing dynamic MRI reconstruction , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[10]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[11]  Tapani Raiko,et al.  Gaussian-Bernoulli deep Boltzmann machine , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[12]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Prateek Jain,et al.  Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization , 2013, SIAM J. Optim..

[14]  Michael Elad,et al.  Sparse Representation for Color Image Restoration , 2008, IEEE Transactions on Image Processing.

[15]  W. Marsden I and J , 2012 .

[16]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[17]  Thomas S. Huang,et al.  Supervised translation-invariant sparse coding , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Kyunghyun Cho,et al.  Simple Sparsification Improves Sparse Denoising Autoencoders in Denoising Highly Corrupted Images , 2013, ICML.

[19]  J. Eggert,et al.  Sparse coding and NMF , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[20]  Jean Ponce,et al.  Learning mid-level features for recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  Lei Zhang,et al.  Metaface learning for sparse representation based face recognition , 2010, 2010 IEEE International Conference on Image Processing.

[22]  Dattatray V. Jadhav,et al.  Feature extraction using Radon and wavelet transforms with application to face recognition , 2009, Neurocomputing.

[23]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[24]  Alain Rakotomamonjy,et al.  Applying alternating direction method of multipliers for constrained dictionary learning , 2013, Neurocomputing.

[25]  Prateek Jain,et al.  Low-rank matrix completion using alternating minimization , 2012, STOC '13.

[26]  Guillermo Sapiro,et al.  Discriminative learned dictionaries for local image analysis , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[28]  H. Bourlard,et al.  Auto-association by multilayer perceptrons and singular value decomposition , 1988, Biological Cybernetics.

[29]  Martial Hebert,et al.  Discriminative Sparse Image Models for Class-Specific Edge Detection and Image Interpretation , 2008, ECCV.

[30]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[31]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[32]  Lei Wang,et al.  Classification of clouds in satellite imagery using over-complete dictionary via sparse representation , 2014, Pattern Recognit. Lett..

[33]  David Zhang,et al.  Fisher Discrimination Dictionary Learning for sparse representation , 2011, 2011 International Conference on Computer Vision.

[34]  Aditya Bhaskara,et al.  More Algorithms for Provable Dictionary Learning , 2014, ArXiv.

[35]  Rabab Kreidieh Ward,et al.  Robust Classifiers for Data Reduced via Random Projections , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[36]  Michael Elad,et al.  Analysis K-SVD: A Dictionary-Learning Algorithm for the Analysis Sparse Model , 2013, IEEE Transactions on Signal Processing.

[37]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[38]  Brendan J. Frey,et al.  k-Sparse Autoencoders , 2013, ICLR.

[39]  KyungHyun Cho,et al.  Simple Sparsification Improves Sparse Denoising Autoencoders in Denoising Highly Noisy Images , 2013 .

[40]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[41]  Larry S. Davis,et al.  Learning a discriminative dictionary for sparse coding via label consistent K-SVD , 2011, CVPR 2011.

[42]  Chang-Hwan Son,et al.  Local Learned Dictionaries Optimized to Edge Orientation for Inverse Halftoning , 2014, IEEE Transactions on Image Processing.

[43]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[44]  Guillermo Sapiro,et al.  Classification and clustering via dictionary learning with structured incoherence and shared features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  Rong Jin,et al.  Unifying discriminative visual codebook generation with classifier training for object category recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[47]  Mike E. Davies,et al.  Dictionary Learning for Sparse Approximations With the Majorization Method , 2009, IEEE Transactions on Signal Processing.

[48]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[49]  Ruimin Shen,et al.  Sparse Group Restricted Boltzmann Machines , 2010, AAAI.

[50]  Ke Huang,et al.  Sparse Representation for Signal Classification , 2006, NIPS.

[51]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.

[52]  Ting Wang,et al.  Kernel Sparse Representation-Based Classifier , 2012, IEEE Transactions on Signal Processing.

[53]  Guillermo Sapiro,et al.  Supervised Dictionary Learning , 2008, NIPS.

[54]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[55]  Zhong Jin,et al.  Kernel sparse representation based classification , 2012, Neurocomputing.

[56]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[57]  Baoxin Li,et al.  Discriminative K-SVD for dictionary learning in face recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[58]  Yoshua Bengio,et al.  Classification using discriminative restricted Boltzmann machines , 2008, ICML '08.

[59]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[60]  Friedrich T. Sommer,et al.  When Can Dictionary Learning Uniquely Recover Sparse Data From Subsamples? , 2011, IEEE Transactions on Information Theory.

[61]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[62]  Min-Sung Koh,et al.  Turbo inpainting: Iterative K-SVD with a new dictionary , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[63]  Ali Aghagolzadeh,et al.  Feature extraction using discrete cosine transform and discrimination power analysis with a face recognition technology , 2010, Pattern Recognit..

[64]  A. Majumdar,et al.  Fast group sparse classification , 2009, Canadian Journal of Electrical and Computer Engineering.

[65]  Daniel Rueckert,et al.  Dictionary Learning and Time Sparsity for Dynamic MR Data Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[66]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.