Eight open questions in the computational modeling of higher sensory cortex

Propelled by advances in biologically inspired computer vision and artificial intelligence, the past five years have seen significant progress in using deep neural networks to model response patterns of neurons in visual cortex. In this paper, we briefly review this progress and then discuss eight key 'open questions' that we believe will drive research in computational models of sensory systems over the next five years, both in visual cortex and beyond.

[1]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[2]  Doris Y. Tsao,et al.  Functional Compartmentalization and Viewpoint Generalization Within the Macaque Face-Processing System , 2010, Science.

[3]  Ha Hong,et al.  Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream , 2013, NIPS.

[4]  H. B. Barlow,et al.  Possible Principles Underlying the Transformations of Sensory Messages , 2012 .

[5]  T. Poggio,et al.  What and where: A Bayesian inference theory of attention , 2010, Vision Research.

[6]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[7]  David J. Freedman,et al.  Categorical representation of visual stimuli in the primate prefrontal cortex. , 2001, Science.

[8]  Nikolaus Kriegeskorte,et al.  Comparison of multivariate classifiers and response normalizations for pattern-information fMRI , 2010, NeuroImage.

[9]  J. Gallant,et al.  A Three-Dimensional Spatiotemporal Receptive Field Model Explains Responses of Area MT Neurons to Naturalistic Movies , 2011, The Journal of Neuroscience.

[10]  N. Kanwisher,et al.  Domain specificity in visual cortex. , 2006, Cerebral cortex.

[11]  K. Norman,et al.  Neural Differentiation Tracks Improved Recall of Competing Memories Following Interleaved Study and Retrieval Practice. , 2015, Cerebral cortex.

[12]  Eero P. Simoncelli,et al.  Metamers of the ventral stream , 2011, Nature Neuroscience.

[13]  T. Poggio,et al.  A Bayesian inference theory of attention: neuroscience and algorithms , 2009 .

[14]  Christopher D. Harvey,et al.  Choice-specific sequences in parietal cortex during a virtual-navigation decision task , 2012, Nature.

[15]  D. C. Essen,et al.  Neural responses to polar, hyperbolic, and Cartesian gratings in area V4 of the macaque monkey. , 1996, Journal of neurophysiology.

[16]  David D. Cox,et al.  Opinion TRENDS in Cognitive Sciences Vol.11 No.8 Untangling invariant object recognition , 2022 .

[17]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[20]  C. Connor,et al.  Curvature processing dynamics in macaque area V4. , 2013, Cerebral cortex.

[21]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[22]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[23]  Charles E Connor,et al.  Underlying principles of visual shape selectivity in posterior inferotemporal cortex , 2004, Nature Neuroscience.

[24]  R. Kiani,et al.  Microstimulation of inferotemporal cortex influences face categorization , 2006, Nature.

[25]  J. Rauschecker,et al.  Cortical Representation of Natural Complex Sounds: Effects of Acoustic Features and Auditory Object Category , 2010, The Journal of Neuroscience.

[26]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[27]  Liam Paninski,et al.  Statistical models for neural encoding, decoding, and optimal stimulus design. , 2007, Progress in brain research.

[28]  R. Malach,et al.  The topography of high-order human object areas , 2002, Trends in Cognitive Sciences.

[29]  Rebecca F. Schwarzlose,et al.  Separate Face and Body Selectivity on the Fusiform Gyrus , 2005, The Journal of Neuroscience.

[30]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[31]  Bevil R. Conway,et al.  Parallel, multi-stage processing of colors, faces and shapes in macaque inferior temporal cortex , 2013, Nature Neuroscience.

[32]  Nicolas Pinto,et al.  Why is Real-World Visual Object Recognition Hard? , 2008, PLoS Comput. Biol..

[33]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[34]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[35]  J. DiCarlo,et al.  Optogenetic and pharmacological suppression of spatial clusters of face neurons reveal their causal role in face gender discrimination , 2015, Proceedings of the National Academy of Sciences.

[36]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[37]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[38]  Charles E. Connor,et al.  A Channel for 3D Environmental Shape in Anterior Inferotemporal Cortex , 2014, Neuron.

[39]  Matthew T. Kaufman,et al.  A neural network that finds a naturalistic solution for the production of muscle activity , 2015, Nature Neuroscience.

[40]  Josh H. McDermott,et al.  Cortical Pitch Regions in Humans Respond Primarily to Resolved Harmonics and Are Located in Specific Tonotopic Regions of Anterior Auditory Cortex , 2013, The Journal of Neuroscience.

[41]  Nicole C. Rust,et al.  Selectivity and Tolerance (“Invariance”) Both Increase as Visual Information Propagates from Cortical Area V4 to IT , 2010, The Journal of Neuroscience.

[42]  Eric T. Carlson,et al.  A neural code for three-dimensional object shape in macaque inferotemporal cortex , 2008, Nature Neuroscience.

[43]  Peter Lennie,et al.  Coding of color and form in the geniculostriate visual pathway (invited review). , 2005, Journal of the Optical Society of America. A, Optics, image science, and vision.

[44]  J. Movshon,et al.  Spatial summation in the receptive fields of simple cells in the cat's striate cortex. , 1978, The Journal of physiology.

[45]  Daniel L. K. Yamins,et al.  Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition , 2014, PLoS Comput. Biol..

[46]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[47]  K. M. Dallenbach.  A puzzle-picture with a new principle of concealment. , 1951, The American journal of psychology.

[48]  Alexander Mordvintsev,et al.  Inceptionism: Going Deeper into Neural Networks , 2015 .

[49]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[50]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[51]  Nicole C. Rust,et al.  Signals in inferotemporal and perirhinal cortex suggest an “untangling” of visual target information , 2013, Nature Neuroscience.

[52]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.

[53]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[54]  P. H. Schiller.  Effect of lesions in visual cortical area V4 on the recognition of transformed objects , 1995, Nature.

[55]  J. Reynolds,et al.  Trade-off between curvature tuning and position invariance in visual area V4 , 2013, Proceedings of the National Academy of Sciences.

[56]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[57]  A. Leventhal,et al.  Signal timing across the macaque visual system. , 1998, Journal of neurophysiology.

[58]  Leon A. Gatys,et al.  Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks , 2015, ArXiv.

[59]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[60]  Sergey Levine,et al.  Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models , 2015, ArXiv.

[61]  Anitha Pasupathy,et al.  Transformation of shape information in the ventral pathway , 2007, Current Opinion in Neurobiology.

[62]  Ha Hong,et al.  Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance , 2015, The Journal of Neuroscience.