Lower bounds on the redundancy of natural images

The light intensities of natural images exhibit a high degree of redundancy. Knowing the exact amount of their statistical dependencies is important for biological vision as well as compression and coding applications but estimating the total amount of redundancy, the multi-information, is intrinsically hard. The common approach is to estimate the multi-information for patches of increasing sizes and divide by the number of pixels. Here, we show that the limiting value of this sequence--the multi-information rate--can be better estimated by using another limiting process based on measuring the mutual information between a pixel and a causal neighborhood of increasing size around it. Although in principle this method has been known for decades, its superiority for estimating the multi-information rate of natural images has not been fully exploited yet. Either method provides a lower bound on the multi-information rate, but the mutual information based sequence converges much faster to the multi-information rate than the conventional method does. Using this fact, we provide improved estimates of the multi-information rate of natural images and a better understanding of its underlying spatial structure.

[1]  Yuhong Yang Elements of Information Theory (2nd ed.). Thomas M. Cover and Joy A. Thomas , 2008 .

[2]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[3]  Eero P. Simoncelli,et al.  Natural signal statistics and sensory gain control , 2001, Nature Neuroscience.

[4]  Eero P. Simoncelli,et al.  Reducing statistical dependencies in natural signals using radial Gaussianization , 2008, NIPS.

[5]  J. Bernardo Expected Information as Expected Utility , 1979 .

[6]  Terrence J. Sejnowski,et al.  Learning Overcomplete Representations , 2000, Neural Computation.

[7]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[8]  Martin J. Wainwright,et al.  Scale Mixtures of Gaussians and the Statistics of Natural Images , 1999, NIPS.

[9]  M. Bethge Factorial coding of natural images: how effective are linear models in removing higher-order dependencies? , 2006, Journal of the Optical Society of America. A, Optics, image science, and vision.

[10]  Matthias Bethge,et al.  The Conjoint Effect of Divisive Normalization and Orientation Selectivity on Redundancy Reduction , 2008, NIPS.

[11]  Hans Föllmer,et al.  On entropy and information gain in random fields , 1973 .

[12]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[13]  Y. Petrov,et al.  Local correlations, information redundancy, and sufficient pixel depth in natural images. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[14]  Matthias Bethge,et al.  Natural Image Coding in V1: How Much Use Is Orientation Selectivity? , 2008, PLoS Comput. Biol..

[15]  R. Jennrich,et al.  Acceleration of the EM Algorithm by using Quasi‐Newton Methods , 1997 .

[16]  William F. Schreiber,et al.  The measurement of third order probability distributions of television signals , 1956, IRE Trans. Inf. Theory.

[17]  G. Buchsbaum,et al.  Trichromacy, opponent colours coding and optimum colour information transmission in the retina , 1983, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[18]  Joseph J. Atick,et al.  What Does the Retina Know about Natural Scenes? , 1992, Neural Computation.

[19]  Albert Perez Ε-admissible Simplifications of the Dependence Structure of a Set of Random Variables , 1977, Kybernetika.

[20]  T. W. Lee,et al.  Chromatic structure of natural scenes. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[21]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[22]  D. W. Scott On optimal and data based histograms , 1979 .

[23]  William Bialek,et al.  Statistics of Natural Images: Scaling in the Woods , 1993, NIPS.

[24]  Michael S. Lewicki,et al.  Emergence of complex cell properties by learning to generalize in natural scenes , 2009, Nature.

[25]  S. Laughlin,et al.  Predictive coding: a fresh view of inhibition in the retina , 1982, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[26]  Alex Pentland,et al.  Discriminative, generative and imitative learning , 2002 .

[27]  D. Field,et al.  Estimates of the information content and dimensionality of natural scenes from proximity distributions. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[28]  T. Sejnowski,et al.  Color opponency is an efficient representation of spectral properties in natural scenes , 2002, Vision Research.

[29]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[30]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[31]  Bruno A. Olshausen,et al.  PROBABILISTIC FRAMEWORK FOR THE ADAPTATION AND COMPARISON OF IMAGE CODES , 1999 .

[32]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[33]  R Linsker,et al.  Perceptual neural organization: some approaches based on network models and information theory. , 1990, Annual review of neuroscience.