Application of concave/Schur-concave functions to the learning of overcomplete dictionaries and sparse representations

Given a very overcomplete m/spl times/n dictionary of representation vectors a/sub i/, A=[a/sub 1/,...,a/sub n/], n/spl Gt/m, an environmentally generated signal, y, can be succinctly represented within the dictionary by obtaining a sparse solution, x, to the linear inverse problem Ax/spl ap/y using various previously proposed methodologies. In particular, sparse solutions can be found by an appropriately regularized minimization of the error e=y-Ax. In this paper we briefly discuss our investigations into the use of concave/Schur-concave functions as regularizing sparsity measures, and their application to the problem of obtaining sparse representations, x, of environmentally generated signals y, and the problem of learning environmentally adapted overcomplete dictionaries.

[1]  Satosi Watanabe,et al.  Pattern recognition as a quest for minimum entropy , 1981, Pattern Recognit..

[2]  Ronald R. Coifman,et al.  Entropy-based algorithms for best basis selection , 1992, IEEE Trans. Inf. Theory.

[3]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[4]  D. Ruderman.  The statistics of natural images , 1994 .

[5]  D. Donoho.  On Minimum Entropy Segmentation , 1994 .

[6]  J. Nadal,et al.  Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer Network 5 , 1994 .

[7]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[8]  Richard C. Puetter,et al.  Pixons and Bayesian image reconstruction , 1994, Optics & Photonics.

[9]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[10]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[11]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[12]  Zhenyu Zhou,et al.  Approximate maximum likelihood hyperparameter estimation for Gibbs priors , 1995, Proceedings., International Conference on Image Processing.

[13]  G. Deco,et al.  An Information-Theoretic Approach to Neural Computing , 1997, Perspectives in Neural Computing.

[14]  Barak A. Pearlmutter,et al.  Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA , 1996, NIPS.

[15]  Dinh-Tuan Pham,et al.  Blind separation of instantaneous mixture of sources via an independent component analysis , 1996, IEEE Trans. Signal Process..

[16]  Richard C. Puetter.  Information, language, and pixon-based image reconstruction , 1996, Optics & Photonics.

[17]  D. Field,et al.  Natural image statistics and efficient coding. , 1996, Network.

[18]  B. Juang,et al.  Selective feature extraction via signal decomposition , 1997, IEEE Signal Process. Lett..

[19]  Ryotaro Kamimura.  Information Controller to Maximize and Minimize Information , 1997, Neural Computation.

[20]  Terrence J. Sejnowski,et al.  Learning Nonlinear Overcomplete Representations for Efficient Coding , 1997, NIPS.

[21]  K. Kreutz-Delgado,et al.  Deriving algorithms for computing sparse solutions to linear inverse problems , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[22]  Bhaskar D. Rao,et al.  Towards a synergistic multistage speech coder , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[23]  Bhaskar D. Rao,et al.  Measures and algorithms for best basis selection , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[24]  Kenji Okajima,et al.  The Gabor function extracts the maximum information from input local signals , 1998, Neural Networks.

[25]  K. Kreutz-Delgado,et al.  Basis selection in the presence of noise , 1998, Conference Record of Thirty-Second Asilomar Conference on Signals, Systems and Computers (Cat. No.98CH36284).

[26]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[27]  S. J. Roberts,et al.  Independent Component Analysis: Source Assessment Separation, a Bayesian Approach , 1998 .

[28]  Bhaskar D. Rao,et al.  Sparse basis selection, ICA, and majorization: towards a unified perspective , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[29]  Bhaskar D. Rao,et al.  An affine scaling methodology for best basis selection , 1999, IEEE Trans. Signal Process..