Predictive Coding and the Slowness Principle: An Information-Theoretic Approach

Understanding the guiding principles of sensory coding strategies is a main goal in computational neuroscience. Among others, the principles of predictive coding and slowness appear to capture aspects of sensory processing. Predictive coding postulates that sensory systems are adapted to the structure of their input signals such that information about future inputs is encoded. Slow feature analysis (SFA) is a method for extracting slowly varying components from quickly varying input signals, thereby learning temporally invariant features. Here, we use the information bottleneck method to state an information-theoretic objective function for temporally local predictive coding. We then show that the linear case of SFA can be interpreted as a variant of predictive coding that maximizes the mutual information between the current output of the system and the input signal in the next time step. This demonstrates that the slowness principle and predictive coding are intimately related.

[1]  Terrence J. Sejnowski,et al.  Slow Feature Analysis: Unsupervised Learning of Invariances , 2002, Neural Computation.

[2]  D. Ballard,et al.  Unifying perception and curiosity , 2006 .

[3]  Naftali Tishby,et al.  Document clustering using word clusters via the information bottleneck method , 2000, SIGIR '00.

[4]  F. Attneave.  Some informational aspects of visual perception. , 1954, Psychological review.

[5]  S. Laughlin,et al.  Predictive coding: a fresh view of inhibition in the retina , 1982, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[6]  Peter Földiák,et al.  Learning Invariance from Transformation Sequences , 1991, Neural Comput..

[7]  Simon B. Eickhoff,et al.  Effects of timing and movement uncertainty implicate the temporo-parietal junction in the prediction of forthcoming motor actions , 2009, NeuroImage.

[8]  Minami Ito,et al.  Size and position invariance of neuronal responses in monkey inferotemporal cortex. , 1995, Journal of neurophysiology.

[9]  A. U.S.,et al.  Predictability , Complexity , and Learning , 2002 .

[10]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[11]  Laurenz Wiskott,et al.  Slow feature analysis yields a rich repertoire of complex cell properties. , 2005, Journal of vision.

[12]  Julian Eggert,et al.  Learning viewpoint invariant object representations using a temporal coherence principle , 2005, Biological Cybernetics.

[13]  J. Magnus,et al.  Matrix Differential Calculus with Applications in Statistics and Econometrics , 1991 .

[14]  Laurenz Wiskott,et al.  What Is the Relation Between Slow Feature Analysis and Independent Component Analysis? , 2006, Neural Computation.

[15]  Naftali Tishby,et al.  Multivariate Information Bottleneck , 2001, Neural Computation.

[16]  Aapo Hyvärinen,et al.  Simple-Cell-Like Receptive Fields Maximize Temporal Coherence in Natural Video , 2003, Neural Computation.

[17]  Néstor Parga,et al.  Redundancy Reduction and Independent Component Analysis: Conditions on Cumulants and Adaptive Approaches , 1997, Neural Computation.

[18]  Terrence J. Sejnowski,et al.  Soft Mixer Assignment in a Hierarchical Generative Model of Natural Scene Statistics , 2006, Neural Computation.

[19]  A. Dimitrov,et al.  Neural coding and decoding: communication channels and quantization , 2001, Network.

[20]  James V. Stone,et al.  A learning rule for extracting spatio-temporal invariances , 1995 .

[21]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[22]  Naftali Tishby,et al.  Past-future information bottleneck in dynamical systems. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[24]  M. Meister,et al.  Dynamic predictive coding by the retina , 2005, Nature.

[25]  Mark H. Johnson,et al.  Object Recognition and Sensitive Periods: A Computational Analysis of Visual Imprinting , 1994, Neural Computation.

[26]  E. Rolls,et al.  INVARIANT FACE AND OBJECT RECOGNITION IN THE VISUAL SYSTEM , 1997, Progress in Neurobiology.

[27]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[28]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[29]  Naftali Tishby,et al.  Extraction of relevant speech features using the information bottleneck method , 2005, INTERSPEECH.

[30]  Gal Chechik,et al.  Information Bottleneck for Gaussian Variables , 2003, J. Mach. Learn. Res..

[31]  Michael S. Lewicki,et al.  A Hierarchical Bayesian Model for Learning Nonlinear Statistical Regularities in Nonstationary Natural Signals , 2005, Neural Computation.

[32]  F.A.A. Kingdom,et al.  The perceived orientation of aliased lines , 1995, Vision Research.

[33]  P. König,et al.  A Model of the Ventral Visual System Based on Temporal Stability and Local Memory , 2006, PLoS biology.

[34]  Laurenz Wiskott,et al.  Slowness and Sparseness Lead to Place, Head-Direction, and Spatial-View Cells , 2007, PLoS Comput. Biol..