Divisive Normalization: Justification and Effectiveness as Efficient Coding Transform

Divisive normalization (DN) has been advocated as an effective nonlinear efficient coding transform for natural sensory signals with applications in biology and engineering. In this work, we aim to establish a connection between the DN transform and the statistical properties of natural sensory signals. Our analysis is based on the use of multivariate t model to capture some important statistical properties of natural sensory signals. The multivariate t model justifies DN as an approximation to the transform that completely eliminates its statistical dependency. Furthermore, using the multivariate t model and measuring statistical dependency with multi-information, we can precisely quantify the statistical dependency that is reduced by the DN transform. We compare this with the actual performance of the DN transform in reducing statistical dependencies of natural sensory signals. Our theoretical analysis and quantitative evaluations confirm DN as an effective efficient coding transform for natural sensory signals. On the other hand, we also observe a previously unreported phenomenon that DN may increase statistical dependencies when the size of pooling is small.

[1]  Matthias Bethge,et al.  The Conjoint Effect of Divisive Normalization and Orientation Selectivity on Redundancy Reduction , 2008, NIPS.

[2]  M. Carandini,et al.  Functional Mechanisms Shaping Lateral Geniculate Responses to Artificial and Natural Stimuli , 2008, Neuron.

[3]  M. Studený,et al.  The Multiinformation Function as a Tool for Measuring Stochastic Dependence , 1998, Learning in Graphical Models.

[4]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[5]  H. B. Barlow,et al.  Possible Principles Underlying the Transformations of Sensory Messages , 2012 .

[6]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[7]  A. McNeil.  Multivariate t Distributions and Their Applications , 2006 .

[8]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[9]  J. M. Foley,et al.  Human luminance pattern-vision mechanisms: masking experiments require a new model. , 1994, Journal of the Optical Society of America. A, Optics, image science, and vision.

[10]  D. F. Andrews,et al.  Scale Mixtures of Normal Distributions , 1974 .

[11]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[12]  D. Heeger.  Normalization of cell responses in cat striate cortex , 1992, Visual Neuroscience.

[13]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[14]  J A Solomon,et al.  Model of visual contrast gain control and pattern masking. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[15]  Eero P. Simoncelli,et al.  Image compression via joint statistical characterization in the wavelet domain , 1999, IEEE Trans. Image Process..

[16]  Eero P. Simoncelli,et al.  Natural image statistics and divisive normalization: Modeling nonlinearity and adaptation in cortical neurons , 2002 .

[17]  C. Enroth-Cugell,et al.  Chapter 9 Visual adaptation and retinal gain controls , 1984 .

[18]  Eero P. Simoncelli,et al.  Nonlinear image representation for efficient perceptual coding , 2006, IEEE Transactions on Image Processing.

[19]  M. Bethge.  Factorial coding of natural images: how effective are linear models in removing higher-order dependencies? , 2006, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  R. Navarro,et al.  Input–output statistical independence in divisive normalization models of V1 neurons , 2003, Network.

[21]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[22]  Oldrich A Vasicek,et al.  A Test for Normality Based on Sample Entropy , 1976 .

[23]  Michael S. Lewicki,et al.  Efficient coding of natural sounds , 2002, Nature Neuroscience.

[24]  Eero P. Simoncelli,et al.  Embedded wavelet image compression based on a joint probability model , 1997, Proceedings of International Conference on Image Processing.

[25]  Jose A. Costa,et al.  On Solutions to Multivariate Maximum α-Entropy Problems , 2003 .

[26]  Liam Paninski,et al.  Estimation of Entropy and Mutual Information , 2003, Neural Computation.

[27]  Geoffrey E. Hinton,et al.  Learning Sparse Topographic Representations with Products of Student-t Distributions , 2002, NIPS.

[28]  Eero P. Simoncelli,et al.  Natural signal statistics and sensory gain control , 2001, Nature Neuroscience.

[29]  Martin J. Wainwright,et al.  Scale Mixtures of Gaussians and the Statistics of Natural Images , 1999, NIPS.

[30]  Bernhard Wegmann,et al.  Statistical dependence between orientation filter outputs used in a human-vision-based image code , 1990, Other Conferences.

[31]  Eero P. Simoncelli,et al.  Nonlinear image representation using divisive normalization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Eero P. Simoncelli,et al.  Nonlinear Extraction of Independent Components of Natural Images Using Radial Gaussianization , 2009, Neural Computation.

[33]  F. Attneave.  Some informational aspects of visual perception. , 1954, Psychological review.

[34]  J. H. van Hateren,et al.  Modelling the Power Spectra of Natural Images: Statistics and Information , 1996, Vision Research.