A geometric approach to sparse coding yields insight into nonlinear responses

In artificial and biological networks, it is a common accepted practice to describe a neurons (biological or artificial) response properties by a two-dimensional feature map (receptive field). However, real neurons have nonlinear response properties which are not represented by their receptive fields. The efficient coding mechanisms such as sparse coding network or ICA, learn the response properties of V1 neurons from natural images using neural networks. These networks learn the receptive fields which are similar to the receptive fields of V1 neurons. These networks also produces some of the nonlinearities (such as end-stopping and non-classical surround effect), which are exhibited by V1 neurons. Here we provide a geometric characterization of these nonlinearities in sparse coding networks. This geometric characterization provides more description about a neuron’s nonlinear response properties than its receptive field. We believe this approach can provide a deeper understanding of how and why sparse representation gives rise to nonlinear responses in V1 neurons. In this study, we consider the iso-response contours of a population of neurons that provide an efficient overcomplete representation of natural scenes. For linear neurons (and those with some simple nonlinearities), the isocontours will be straight and orthogonal to the receptive field vector. However, with an overcomplete code, the lack of orthogonality between encoding vectors can result in significant redundancy. As previously suggested, positive curvature of the isocontours can remove much of this redundancy. Here we demonstrate that this curvature also produces a family of nonlinearities found in V1 neurons. We also examine the nonlinear response of the sparse coding network with this approach. We trained a sparse coding network on natural scenes and calculated the isocontours of each neuron by measuring the response of the network over a uniform grid of points in the stimulus space. We find the isocontours are curved toward the encoding vectors, and that the curvature varies as a function of the sparse prior and degrees of sparsity and overcompleteness. In order to investigate the state space response of vectors in the sparse coding network, we first examined its response in a two-dimensional state space for data with three causes. Once the encoding vectors were determined, the isocontours were found by calculating the response of the network for points over a uniform grid in state space. We demonstrate the change in iso-respnse contour curvature for highdimensional natural scene data with respect to different sparse priors, degree of sparsity and overcompleteness. We also show how invariance fits into this framework and describe how the curvature depends on the relative position of the vectors and the cost functions used in the network.