Sparsity and Nullity: Paradigms for Analysis Dictionary Learning

Sparse models in dictionary learning have been successfully applied in a wide variety of machine learning and computer vision problems, and as a result have recently attracted increased research interest. Another interesting related problem based on linear equality constraints, namely the sparse null space (SNS) problem, first appeared in 1986 and has since inspired results on sparse basis pursuit. In this paper, we investigate the relation between the SNS problem and the analysis dictionary learning (ADL) problem, and show that the SNS problem plays a central role, and may be utilized to solve dictionary learning problems. Moreover, we propose an efficient algorithm of sparse null space basis pursuit (SNS-BP) and extend it to a solution of ADL. Experimental results on numerical synthetic data and real-world data are further presented to validate the performance of our method.

[1]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[2]  Michael Elad,et al.  Analysis K-SVD: A Dictionary-Learning Algorithm for the Analysis Sparse Model , 2013, IEEE Transactions on Signal Processing.

[3]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[4]  T. Coleman,et al.  The null space problem II. Algorithms , 1987 .

[5]  Yue Liu,et al.  Texture Classification Based on Completed Modeling of Local Binary Pattern , 2011, 2011 International Conference on Computational and Information Sciences.

[6]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[7]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[8]  Michael Elad,et al.  Sparse and Redundant Representation Modeling—What Next? , 2012, IEEE Signal Processing Letters.

[9]  Trygve Randen,et al.  Filtering for Texture Classification: A Comparative Study , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Guillermo Sapiro,et al.  Supervised Dictionary Learning , 2008, NIPS.

[11]  Yunjin Chen,et al.  Insights Into Analysis Operator Learning: From Patch-Based Sparse Models to Higher Order MRFs , 2014, IEEE Transactions on Image Processing.

[12]  Alex Pothen,et al.  Sparse null basis computations in structural optimization , 1989 .

[13]  Zheng Yang,et al.  Locality-Constrained Concept Factorization , 2011, IJCAI.

[14]  Ehsan Elhamifar,et al.  Sparse subspace clustering , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  J. Gilbert.  Computing a sparse basis for the null space , 1987 .

[16]  Hamid Krim,et al.  Optimal Operator Space Pursuit: A Framework for Video Sequence Data Analysis , 2012, ACCV.

[17]  E. Candès.  The restricted isometry property and its implications for compressed sensing , 2008 .

[18]  Klaus Diepold,et al.  Analysis Operator Learning and its Application to Image Reconstruction , 2012, IEEE Transactions on Image Processing.

[19]  Michael Elad,et al.  K-SVD dictionary-learning for the analysis sparse model , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  V. Climenhaga.  Markov chains and mixing times , 2013 .

[21]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[22]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[23]  T. Blumensath,et al.  Theory and Applications , 2011 .

[24]  Hamid Krim,et al.  Robust Subspace Recovery via Bi-Sparsity Pursuit , 2018 .

[25]  Michael Elad,et al.  Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries , 2006, IEEE Transactions on Image Processing.

[26]  T. Coleman,et al.  The null space problem I. complexity , 1986 .

[27]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[28]  Michael Elad,et al.  On the Role of Sparse and Redundant Representations in Image Processing , 2010, Proceedings of the IEEE.

[29]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[30]  Rémi Gribonval,et al.  Analysis operator learning for overcomplete cosparse representations , 2011, 2011 19th European Signal Processing Conference.

[31]  Ke Huang,et al.  Sparse Representation for Signal Classification , 2006, NIPS.

[32]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[33]  Yoram Bresler,et al.  Structured Overcomplete Sparsifying Transform Learning with Convergence Guarantees and Applications , 2015, International Journal of Computer Vision.

[34]  Rémi Gribonval,et al.  Constrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling , 2012, IEEE Transactions on Signal Processing.

[35]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[36]  Léon Bottou,et al.  The Tradeoffs of Large Scale Learning , 2007, NIPS.

[37]  Liyi Dai,et al.  Sparse null space basis pursuit and analysis dictionary learning for high-dimensional data analysis , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).