Learning and storing the parts of objects: IMF

A central concern for many learning algorithms is how to efficiently store what the algorithm has learned. An algorithm for the compression of Nonnegative Matrix Factorizations is presented. Compression is achieved by embedding the factorization in an encoding routine. Its performance is investigated using two standard test images, Peppers and Barbara. The compression ratio (18:1) achieved by the proposed Matrix Factorization improves the storage-ability of Nonnegative Matrix Factorizations without significantly degrading accuracy (≈ 1-3dB degradation is introduced). We learn as before, but storage is cheaper.

[1]  A. Bruckstein,et al.  Sparse non-negative solution of a linear system of equations is unique , 2008, 2008 3rd International Symposium on Communications, Control and Signal Processing.

[2]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[3]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[4]  Terry A. Welch,et al.  A Technique for High-Performance Data Compression , 1984, Computer.

[5]  Shimon Even,et al.  An algorithm for optimal prefix parsing of a noiseless and memoryless channel , 1973, IEEE Trans. Inf. Theory.

[6]  Frederick Jelinek,et al.  On variable-length-to-block coding , 1972, IEEE Trans. Inf. Theory.

[7]  Daniel W. E. Schobben,et al.  Dither and data compression , 1997, IEEE Trans. Signal Process..

[8]  Ruairí de Fréin.  Ghostbusters: A parts-based NMF algorithm , 2013 .

[9]  Abraham Lempel,et al.  Compression of individual sequences via variable-rate coding , 1978, IEEE Trans. Inf. Theory.

[10]  Jacob Ziv,et al.  Coding of sources with unknown statistics-II: Distortion relative to a fidelity criterion , 1972, IEEE Trans. Inf. Theory.

[11]  Peter Deutsch,et al.  GZIP file format specification version 4.3 , 1996, RFC.

[12]  R. Gallager.  Information Theory and Reliable Communication , 1968 .

[13]  Shimon Even,et al.  Efficient Generation of Optimal Prefix Code: Equiprobable Words Using Unequal Cost Letters , 1975, JACM.

[14]  Richard M. Karp,et al.  Minimum-redundancy coding for the discrete noiseless channel , 1961, IRE Trans. Inf. Theory.

[15]  Patrik O. Hoyer,et al.  Non-negative sparse coding , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[16]  Barak A. Pearlmutter,et al.  Convolutive Non-Negative Matrix Factorisation with a Sparseness Constraint , 2006 .

[17]  Herbert Gish,et al.  Asymptotically efficient quantizing , 1968, IEEE Trans. Inf. Theory.

[18]  Mark D. Plumbley,et al.  Theorems on Positive Data: On the Uniqueness of NMF , 2008, Comput. Intell. Neurosci..

[19]  Ben Varn,et al.  Optimal Variable Length Codes (Arbitrary Symbol Cost and Equal Code Word Probability) , 1971, Inf. Control..

[20]  Barak A. Pearlmutter,et al.  Blind Source Separation by Sparse Decomposition in a Signal Dictionary , 2001, Neural Computation.

[21]  Lee D. Davisson,et al.  Universal noiseless coding , 1973, IEEE Trans. Inf. Theory.

[22]  Ruairí de Fréin,et al.  Learning speech features in the presence of noise: Sparse convolutive robust non-negative matrix factorization , 2009, 2009 16th International Conference on Digital Signal Processing.

[23]  D. Huffman.  A Method for the Construction of Minimum-Redundancy Codes , 1952 .

[24]  Andrzej Cichocki,et al.  When is non-negative matrix decomposition unique? , 2008, 2008 42nd Annual Conference on Information Sciences and Systems.

[25]  Abraham Lempel,et al.  A universal algorithm for sequential data compression , 1977, IEEE Trans. Inf. Theory.