A Personal Introduction to Theoretical Dictionary Learning

I started to get interested in dictionary learning in 2007 at the end of my 2nd PhD year. My PhD topic was roughly sparsity and dictionaries, as this was what Pierre (Vandergheynst), my advisor, made almost all the group do to some degree. Since the group was a happy mix of computer scientists, electric engineers and mathematicians led by a theoretical physicist, a dictionary Φ was defined as a collection of K unit norm vectors φk ∈ Rd called atoms. The atoms were stacked as columns in a matrix, which by abuse of notation was also referred to as the dictionary, that is Φ = (φ1, . . . ,φK) ∈ Rd×K . A signal y ∈ Rd was called sparse in a dictionary Φ if up to a small approximation error or noise it could be represented as linear combination of a small (sparse) number of dictionary atoms,

[1]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[2]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[3]  Huan Wang,et al.  Exact Recovery of Sparsely-Used Dictionaries , 2012, COLT.

[4]  Rémi Gribonval,et al.  Sparse and Spurious: Dictionary Learning With Noise and Outliers , 2014, IEEE Transactions on Information Theory.

[5]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[6]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[7]  Fabian J. Theis,et al.  Sparse component analysis and blind source separation of underdetermined mixtures , 2005, IEEE Transactions on Neural Networks.

[8]  Pierre Vandergheynst,et al.  Dictionary Preconditioning for Greedy Algorithms , 2008, IEEE Transactions on Signal Processing.

[9]  H. Rauhut.  On the Impossibility of Uniform Sparse Reconstruction using Greedy Methods , 2007 .

[10]  Shie Mannor,et al.  The Sample Complexity of Dictionary Learning , 2010, COLT.

[11]  Yonina C. Eldar,et al.  Performance limits of dictionary learning for sparse coding , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[12]  Karin Schnass,et al.  On the Identifiability of Overcomplete Dictionaries via the Minimisation Principle Underlying K-SVD , 2013, ArXiv.

[13]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[14]  Sanjeev Arora,et al.  New Algorithms for Learning Incoherent and Overcomplete Dictionaries , 2013, COLT.

[15]  Mark D. Plumbley.  Dictionary Learning for L1-Exact Sparse Coding , 2007, ICA.

[16]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[17]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[18]  H. Rauhut,et al.  Atoms of All Channels, Unite! Average Case Analysis of Multi-Channel Sparse Recovery Using Greedy Algorithms , 2008 .

[19]  Jean-Jacques Fuchs.  Extension of the Pisarenko method to sparse linear arrays , 1997, IEEE Trans. Signal Process..

[20]  A. Bruckstein,et al.  On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them , 2006 .

[21]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[22]  R. Gribonval,et al.  Basis Identification from Random Sparse Samples , 2009 .

[23]  Rémi Gribonval,et al.  Dictionary identifiability from few training samples , 2008, 2008 16th European Signal Processing Conference.

[24]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.

[25]  Anima Anandkumar,et al.  Exact Recovery of Sparsely Used Overcomplete Dictionaries , 2013, ArXiv.

[26]  Deanna Needell,et al.  Uniform Uncertainty Principle and Signal Recovery via Regularized Orthogonal Matching Pursuit , 2007, Found. Comput. Math..

[27]  Karin Schnass,et al.  Local identification of overcomplete dictionaries , 2014, J. Mach. Learn. Res..

[28]  Rémi Gribonval,et al.  Sample Complexity of Dictionary Learning and Other Matrix Factorizations , 2013, IEEE Transactions on Information Theory.

[29]  R. Gribonval,et al.  Some recovery conditions for basis learning by L1-minimization , 2008, 2008 3rd International Symposium on Communications, Control and Signal Processing.

[30]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[31]  D. Donoho,et al.  Maximal Sparsity Representation via l 1 Minimization , 2002 .

[32]  Marc E. Pfetsch,et al.  The Computational Complexity of the Restricted Isometry Property, the Nullspace Property, and Related Concepts in Compressed Sensing , 2012, IEEE Transactions on Information Theory.

[33]  Kjersti Engan,et al.  Method of optimal directions for frame design , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[34]  Massimiliano Pontil,et al.  $K$ -Dimensional Coding Schemes in Hilbert Spaces , 2010, IEEE Transactions on Information Theory.

[35]  Alexander G. Gray,et al.  On the Sample Complexity of Predictive Sparse Coding , 2012, ArXiv.

[36]  Bhaskar D. Rao,et al.  FOCUSS-based dictionary learning algorithms , 2000, SPIE Optics + Photonics.

[37]  Rémi Gribonval,et al.  Local stability and robustness of sparse dictionary learning in the presence of noise , 2012, ArXiv.

[38]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions with matching pursuit , 1994, Defense, Security, and Sensing.

[39]  Vladimir Temlyakov,et al.  CAMBRIDGE MONOGRAPHS ON APPLIED AND COMPUTATIONAL MATHEMATICS , 2022 .

[40]  Jan Vybíral,et al.  Learning Functions of Few Arbitrary Linear Parameters in High Dimensions , 2010, Found. Comput. Math..

[41]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[42]  J. Tropp.  On the conditioning of random subdictionaries , 2008 .

[43]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[44]  Aditya Bhaskara,et al.  More Algorithms for Provable Dictionary Learning , 2014, ArXiv.

[45]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[46]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[47]  Prateek Jain,et al.  Learning Sparsely Used Overcomplete Dictionaries via Alternating Minimization , 2013, SIAM J. Optim..

[48]  Pierre Vandergheynst,et al.  Compressed Sensing and Redundant Dictionaries , 2007, IEEE Transactions on Information Theory.

[49]  Huan Wang,et al.  On the local correctness of ℓ1-minimization for dictionary learning , 2011, 2014 IEEE International Symposium on Information Theory.

[50]  Pierre Vandergheynst,et al.  Average Performance Analysis for Thresholding , 2007, IEEE Signal Processing Letters.