Sparse deep predictive coding captures contour integration capabilities of the early visual system

Both neurophysiological and psychophysical experiments have pointed out the crucial role of recurrent and feedback connections to process context-dependent information in the early visual cortex. While numerous models have accounted for feedback effects at either neural or representational level, none of them were able to bind those two levels of analysis. Is it possible to describe feedback effects at both levels using the same model? We answer this question by combining Predictive Coding (PC) and Sparse Coding (SC) into a hierarchical and convolutional framework applied to realistic problems. In the Sparse Deep Predictive Coding (SDPC) model, the SC component models the internal recurrent processing within each layer, and the PC component describes the interactions between layers using feedforward and feedback connections. Here, we train a 2-layered SDPC on two different databases of images, and we interpret it as a model of the early visual system (V1 & V2). We first demonstrate that once the training has converged, SDPC exhibits oriented and localized receptive fields in V1 and more complex features in V2. Second, we analyze the effects of feedback on the neural organization beyond the classical receptive field of V1 neurons using interaction maps. These maps are similar to association fields and reflect the Gestalt principle of good continuation. We demonstrate that feedback signals reorganize interaction maps and modulate neural activity to promote contour integration. Third, we demonstrate at the representational level that the SDPC feedback connections are able to overcome noise in input images. Therefore, the SDPC captures the association field principle at the neural level which results in a better reconstruction of blurred images at the representational level.

[1]  Gabriel Cristóbal,et al.  Self-Invertible 2D Log-Gabor Wavelets , 2007, International Journal of Computer Vision.

[2]  Yann LeCun,et al.  Structured sparse coding via lateral inhibition , 2011, NIPS.

[3]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[4]  V. Bringuier,et al.  The visual cortical association field: A Gestalt concept or a psychophysiological entity? , 2000, Journal of Physiology-Paris.

[5]  R. Desimone,et al.  Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. , 1981, Journal of neurophysiology.

[6]  Franck Ruffier,et al.  Effect of Top-Down Connections in Hierarchical Sparse Coding , 2020, Neural Computation.

[7]  C. N. Boehler,et al.  Rapid recurrent processing gates awareness in primary visual cortex , 2008, Proceedings of the National Academy of Sciences.

[8]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[9]  Yann LeCun,et al.  Convolutional Matching Pursuit and Dictionary Training , 2010, ArXiv.

[10]  Beren Millidge,et al.  Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs , 2020, Neural Computation.

[11]  Nikolaus Kriegeskorte,et al.  Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition , 2017, bioRxiv.

[12]  Michael Elad,et al.  Multi Layer Sparse Coding: the Holistic Way , 2018, SIAM J. Math. Data Sci..

[13]  Michael Elad,et al.  Dictionaries for Sparse Representation Modeling , 2010, Proceedings of the IEEE.

[14]  Karl J. Friston Hierarchical Models in the Brain , 2008, PLoS Comput. Biol..

[15]  U. Polat,et al.  Lateral interactions between spatial channels: Suppression and facilitation revealed by lateral masking experiments , 1993, Vision Research.

[16]  H. Neumann,et al.  The Role of Attention in Figure-Ground Segregation in Areas V1 and V4 of the Visual Cortex , 2012, Neuron.

[17]  Karl J. Friston The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[18]  C. Gilbert,et al.  Interactions between attention, context and learning in primary visual cortex , 2000, Vision Research.

[19]  Yann LeCun,et al.  What is the best multi-stage architecture for object recognition? , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[20]  David J. Jilk,et al.  Early recurrent feedback facilitates visual object recognition under challenging conditions , 2014, Front. Psychol..

[21]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[22]  Alain Destexhe,et al.  Suppressive Traveling Waves Shape Representations of Illusory Motion in Primary Visual Cortex of Awake Primate , 2019, The Journal of Neuroscience.

[23]  Richard T Born,et al.  Corticocortical Feedback Contributes to Surround Suppression in V1 of the Alert Primate , 2013, The Journal of Neuroscience.

[24]  Eero P. Simoncelli,et al.  Nonlinear image representation using divisive normalization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[26]  A. Angelucci,et al.  Top-down feedback controls spatial summation and response amplitude in primate visual cortex , 2018, Nature Communications.

[27]  Michael Elad,et al.  Convolutional Neural Networks Analyzed via Convolutional Sparse Coding , 2016, J. Mach. Learn. Res..

[28]  M. Shiffrar,et al.  Different motion sensitive units are involved in recovering the direction of moving lines , 1993, Vision Research.

[29]  P. Roelfsema Cortical algorithms for perceptual grouping. , 2006, Annual review of neuroscience.

[30]  D. Knill,et al.  The Bayesian brain: the role of uncertainty in neural coding and computation , 2004, Trends in Neurosciences.

[31]  Karl J. Friston,et al.  Attention, Uncertainty, and Free-Energy , 2010, Front. Hum. Neurosci..

[32]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[33]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[34]  J. P. Jones,et al.  The two-dimensional spatial structure of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[35]  D. Ringach Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. , 2002, Journal of neurophysiology.

[36]  Yves Frégnac,et al.  Synaptic Correlates of Low-Level Perception in V1 , 2016, The Journal of Neuroscience.

[37]  Y. Frégnac,et al.  The “silent” surround of V1 receptive fields: theory and experiments , 2003, Journal of Physiology-Paris.

[38]  V. Lamme,et al.  The distinct modes of vision offered by feedforward and recurrent processing , 2000, Trends in Neurosciences.

[39]  David J. Field,et al.  Contour integration by the human visual system: Evidence for a local “association field” , 1993, Vision Research.

[40]  Michael Elad,et al.  Multilayer Convolutional Sparse Modeling: Pursuit and Dictionary Learning , 2017, IEEE Transactions on Signal Processing.

[41]  C. Gilbert,et al.  Interactions between feedback and lateral connections in the primary visual cortex , 2017, Proceedings of the National Academy of Sciences.

[42]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[43]  Jeffrey S. Perry,et al.  Edge co-occurrence in natural images predicts contour grouping performance , 2001, Vision Research.

[44]  Eugenio Culurciello,et al.  Deep Predictive Coding Network for Object Recognition , 2018, ICML.

[45]  David J. Jilk,et al.  Recurrent Processing during Object Recognition , 2011, Front. Psychol..

[46]  Klaus Obermayer,et al.  The Role of Lateral Cortical Competition in Ocular Dominance Development , 1998, NIPS.

[47]  Tim Curran,et al.  The Limits of Feedforward Vision: Recurrent Processing Promotes Robust Object Recognition when Objects Are Degraded , 2012, Journal of Cognitive Neuroscience.

[48]  Yali Amit,et al.  Deep Learning With Asymmetric Connections and Hebbian Updates , 2018, Front. Comput. Neurosci..

[49]  Nikola T. Markov,et al.  Weight Consistency Specifies Regularities of Macaque Cortical Networks , 2010, Cerebral cortex.

[50]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Karl J. Friston,et al.  Canonical Microcircuits for Predictive Coding , 2012, Neuron.

[52]  J. Bullier Integrated model of visual processing , 2001, Brain Research Reviews.

[53]  James A. Bednar,et al.  Edge co-occurrences can account for rapid categorization of natural versus animal images , 2015, Scientific Reports.

[54]  B. Julesz,et al.  Perceptual sensitivity maps within globally defined visual shapes , 1994, Nature.

[55]  U. Eysel,et al.  Evidence for a contribution of lateral inhibition to orientation tuning and direction selectivity in cat visual cortex: reversible inactivation of functionally characterized sites combined with neuroanatomical tracing techniques , 1998, The European journal of neuroscience.

[56]  Guillermo Sapiro,et al.  Non-local sparse models for image restoration , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[57]  T. Pham-Gia,et al.  The mean and median absolute deviations , 2001 .

[58]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[59]  H. Wallach On the visually perceived direction of motion ' ' by Hans Wallach : 60 years later , 1997 .

[60]  Johannes Jacobus Fahrenfort,et al.  The spatiotemporal profile of cortical processing leading up to visual perception. , 2008, Journal of vision.

[61]  Stewart Shipp,et al.  Neural Elements for Predictive Coding , 2016, Front. Psychol..

[62]  C. Gilbert,et al.  Adult Visual Cortical Plasticity , 2012, Neuron.

[63]  Michael W. Spratling Unsupervised Learning of Generative and Discriminative Weights Encoding Elementary Image Components in a Predictive Coding Model of Cortical Function , 2012, Neural Computation.

[64]  Eero P. Simoncelli,et al.  How MT cells analyze the motion of visual patterns , 2006, Nature Neuroscience.

[65]  Rajesh P. N. Rao,et al.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. , 1999 .

[66]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[67]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[68]  David J. Freedman,et al.  A Comparison of Primate Prefrontal and Inferior Temporal Cortices during Visual Categorization , 2003, The Journal of Neuroscience.

[69]  Jean Lorenceau,et al.  Orientation dependent modulation of apparent speed: psychophysical evidence , 2002, Vision Research.

[70]  Olaf Sporns,et al.  The small world of the cerebral cortex , 2007, Neuroinformatics.

[71]  S. Laughlin,et al.  Predictive coding: a fresh view of inhibition in the retina , 1982, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[72]  Gabriel Kreiman,et al.  Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning , 2016, ICLR.

[73]  R. Shapley,et al.  “On the Visually Perceived Direction of Motion” by Hans Wallach: 60 Years Later , 1996 .

[74]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[75]  Brendan J. Frey,et al.  k-Sparse Autoencoders , 2013, ICLR.

[76]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[77]  Rafal Bogacz,et al.  An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity , 2017, Neural Computation.

[78]  C. Gilbert,et al.  Top-down influences on visual processing , 2013, Nature Reviews Neuroscience.

[79]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[80]  Pascal Mamassian,et al.  Bayesian modeling of dynamic motion integration , 2007, Journal of Physiology-Paris.

[81]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[82]  Joshua Correll,et al.  The Chicago face database: A free stimulus set of faces and norming data , 2015, Behavior research methods.

[83]  Terrence J. Sejnowski,et al.  Cortical travelling waves: mechanisms and computational principles , 2018, Nature Reviews Neuroscience.

[84]  Siegrid Löwel,et al.  GABA-inactivation attenuates colinear facilitation in cat primary visual cortex , 2002, Experimental Brain Research.

[85]  Laurent U. Perrinet,et al.  Role of Homeostasis in Learning Sparse Representations , 2007, Neural Computation.

[86]  S. Thorpe,et al.  Rapid categorization of natural images by rhesus monkeys , 1998, Neuroreport.

[87]  Michael W. Spratling A Hierarchical Predictive Coding Model of Object Recognition in Natural Images , 2016, Cognitive Computation.

[88]  Joseph Marino,et al.  Predictive Coding, Variational Autoencoders, and Biological Connections , 2019, Neural Computation.

[89]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[90]  Xiaoxia Sun,et al.  Supervised Deep Sparse Coding Networks for Image Classification , 2020, IEEE Transactions on Image Processing.

[91]  N. Kriegeskorte,et al.  Faciotopy—A face-feature map with face-like topology in the human occipital face area , 2015, Cortex.

[92]  Hualou Liang,et al.  Incremental Integration of Global Contours through Interplay between Visual Cortical Areas , 2014, Neuron.

[93]  Claus-Christian Carbon,et al.  Part-to-Whole Effects and Configural Processing in Faces , 2004 .