Sparse dictionary learning from 1-BIT data

This work examines a sparse dictionary learning task - that of fitting a collection of data points, arranged as columns of a matrix, to a union of low-dimensional linear subspaces - in settings where only highly quantized (single bit) observations of the data matrix entries are available. We analyze a complexity penalized maximum likelihood estimation strategy, and obtain finite-sample bounds for the average per-element squared approximation error of the estimate produced by our approach. Our results are reminiscent of traditional parametric estimation tasks - we show here that despite the highly-quantized observations, the normalized per-element estimation error is bounded by the ratio between the number of “degrees of freedom” of the matrix and its dimension.

[1]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.

[2]  P. Grünwald.  The Minimum Description Length Principle (Adaptive Computation and Machine Learning) , 2007 .

[3]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[4]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[5]  Robert D. Nowak,et al.  Signal Reconstruction From Noisy Random Projections , 2006, IEEE Transactions on Information Theory.

[6]  R. Nowak,et al.  Multiscale likelihood analysis and complexity penalized estimation , 2004, math/0406424.

[7]  Rémi Gribonval,et al.  Local stability and robustness of sparse dictionary learning in the presence of noise , 2012, ArXiv.

[8]  Richard G. Baraniuk,et al.  1-Bit compressive sensing , 2008, 2008 42nd Annual Conference on Information Sciences and Systems.

[9]  Yaniv Plan,et al.  Robust 1-bit Compressed Sensing and Sparse Logistic Regression: A Convex Programming Approach , 2012, IEEE Transactions on Information Theory.

[10]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[11]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[12]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[13]  Huan Wang,et al.  On the local correctness of ℓ1-minimization for dictionary learning , 2011, 2014 IEEE International Symposium on Information Theory.

[14]  Zhi-Quan Luo,et al.  Universal decentralized estimation in a bandwidth constrained sensor network , 2005, IEEE Transactions on Information Theory.

[15]  Alejandro Ribeiro,et al.  Bandwidth-constrained distributed estimation for wireless sensor networks-part II: unknown probability density function , 2006, IEEE Transactions on Signal Processing.

[16]  C. Craig.  On the Tchebychef Inequality of Bernstein , 1933 .

[17]  Tong Zhang,et al.  On the Convergence of MDL Density Estimation , 2004, COLT.

[18]  Ewout van den Berg,et al.  1-Bit Matrix Completion , 2012, ArXiv.

[19]  Guillermo Sapiro,et al.  Online dictionary learning for sparse coding , 2009, ICML '09.

[20]  P. Boufounos.  Greedy sparse signal reconstruction from sign measurements , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[21]  Andrew R. Barron,et al.  Mixture Density Estimation , 1999, NIPS.

[22]  Andrew R. Barron,et al.  Complexity Regularization with Application to Artificial Neural Networks , 1991 .

[23]  Karin Schnass,et al.  Dictionary Identification—Sparse Matrix-Factorization via $\ell_1$ -Minimization , 2009, IEEE Transactions on Information Theory.

[24]  Julien Mairal,et al.  Proximal Methods for Sparse Hierarchical Dictionary Learning , 2010, ICML.

[25]  Guillermo Sapiro,et al.  An MDL Framework for Sparse Coding and Dictionary Learning , 2011, IEEE Transactions on Signal Processing.

[26]  Yaniv Plan,et al.  One‐Bit Compressed Sensing by Linear Programming , 2011, ArXiv.

[27]  Andrew R. Barron,et al.  Minimum complexity density estimation , 1991, IEEE Trans. Inf. Theory.

[28]  Alejandro Ribeiro,et al.  Bandwidth-constrained distributed estimation for wireless sensor Networks-part I: Gaussian case , 2006, IEEE Transactions on Signal Processing.

[29]  Richard G. Baraniuk,et al.  Robust support recovery using sparse compressive sensing matrices , 2011, 2011 45th Annual Conference on Information Sciences and Systems.

[30]  Robert D. Nowak,et al.  Sample complexity for 1-bit compressed sensing and sparse classification , 2010, 2010 IEEE International Symposium on Information Theory.

[31]  Stephen P. Boyd,et al.  Compressed Sensing With Quantized Measurements , 2010, IEEE Signal Processing Letters.

[32]  Zhi-Quan Luo.  An isotropic universal decentralized estimation scheme for a bandwidth constrained ad hoc sensor network , 2005, IEEE Journal on Selected Areas in Communications.

[33]  P. Massart,et al.  Risk bounds for model selection via penalization , 1999 .

[34]  Zhi-Quan Luo,et al.  Decentralized estimation in an inhomogeneous sensing environment , 2005, IEEE Transactions on Information Theory.

[35]  Karin Schnass,et al.  On the Identifiability of Overcomplete Dictionaries via the Minimisation Principle Underlying K-SVD , 2013, ArXiv.