Using deep learning to probe the neural code for images in primary visual cortex

Primary visual cortex (V1) is the first stage of cortical image processing, and major effort in systems neuroscience is devoted to understanding how it encodes information about visual stimuli. Within V1, many neurons respond selectively to edges of a given preferred orientation: These are known as either simple or complex cells. Other neurons respond to localized center–surround image features. Still others respond selectively to certain image stimuli, but the specific features that excite them are unknown. Moreover, even for the simple and complex cells—the best-understood V1 neurons—it is challenging to predict how they will respond to natural image stimuli. Thus, there are important gaps in our understanding of how V1 encodes images. To fill this gap, we trained deep convolutional neural networks to predict the firing rates of V1 neurons in response to natural image stimuli, and we find that the predicted firing rates are highly correlated (\begin{document}\newcommand{\bialpha}{\boldsymbol{\alpha}}\newcommand{\bibeta}{\boldsymbol{\beta}}\newcommand{\bigamma}{\boldsymbol{\gamma}}\newcommand{\bidelta}{\boldsymbol{\delta}}\newcommand{\bivarepsilon}{\boldsymbol{\varepsilon}}\newcommand{\bizeta}{\boldsymbol{\zeta}}\newcommand{\bieta}{\boldsymbol{\eta}}\newcommand{\bitheta}{\boldsymbol{\theta}}\newcommand{\biiota}{\boldsymbol{\iota}}\newcommand{\bikappa}{\boldsymbol{\kappa}}\newcommand{\bilambda}{\boldsymbol{\lambda}}\newcommand{\bimu}{\boldsymbol{\mu}}\newcommand{\binu}{\boldsymbol{\nu}}\newcommand{\bixi}{\boldsymbol{\xi}}\newcommand{\biomicron}{\boldsymbol{\micron}}\newcommand{\bipi}{\boldsymbol{\pi}}\newcommand{\birho}{\boldsymbol{\rho}}\newcommand{\bisigma}{\boldsymbol{\sigma}}\newcommand{\bitau}{\boldsymbol{\tau}}\newcommand{\biupsilon}{\boldsymbol{\upsilon}}\newcommand{\biphi}{\boldsymbol{\phi}}\newcommand{\bichi}{\boldsymbol{\chi}}\newcommand{\bipsi}{\boldsymbol{\psi}}\newcommand{\biomega}{\boldsymbol{\omega}}{\overline {{\bf{CC}}} _{{\bf{norm}}}}\end{document} = 0.556 ± 0.01) with the neurons' actual firing rates over a population of 355 neurons. This performance value is quoted for all neurons, with no selection filter. Performance is better for more active neurons: When evaluated only on neurons with mean firing rates above 5 Hz, our predictors achieve correlations of \begin{document}\newcommand{\bialpha}{\boldsymbol{\alpha}}\newcommand{\bibeta}{\boldsymbol{\beta}}\newcommand{\bigamma}{\boldsymbol{\gamma}}\newcommand{\bidelta}{\boldsymbol{\delta}}\newcommand{\bivarepsilon}{\boldsymbol{\varepsilon}}\newcommand{\bizeta}{\boldsymbol{\zeta}}\newcommand{\bieta}{\boldsymbol{\eta}}\newcommand{\bitheta}{\boldsymbol{\theta}}\newcommand{\biiota}{\boldsymbol{\iota}}\newcommand{\bikappa}{\boldsymbol{\kappa}}\newcommand{\bilambda}{\boldsymbol{\lambda}}\newcommand{\bimu}{\boldsymbol{\mu}}\newcommand{\binu}{\boldsymbol{\nu}}\newcommand{\bixi}{\boldsymbol{\xi}}\newcommand{\biomicron}{\boldsymbol{\micron}}\newcommand{\bipi}{\boldsymbol{\pi}}\newcommand{\birho}{\boldsymbol{\rho}}\newcommand{\bisigma}{\boldsymbol{\sigma}}\newcommand{\bitau}{\boldsymbol{\tau}}\newcommand{\biupsilon}{\boldsymbol{\upsilon}}\newcommand{\biphi}{\boldsymbol{\phi}}\newcommand{\bichi}{\boldsymbol{\chi}}\newcommand{\bipsi}{\boldsymbol{\psi}}\newcommand{\biomega}{\boldsymbol{\omega}}{\overline {{\bf{CC}}} _{{\bf{norm}}}}\end{document} = 0.69 ± 0.01 with the neurons' true firing rates. We find that the firing rates of both orientation-selective and non-orientation-selective neurons can be predicted with high accuracy. Additionally, we use a variety of models to benchmark performance and find that our convolutional neural-network model makes more accurate predictions.