Different Goal-driven CNNs Affect Performance of Visual Encoding Models based on Deep Learning

A convolutional neural network with outstanding performance in computer vision can be used to construct an encoding model that simulates the process of human visual information processing. However, training goal of the network may have impacted the performance of encoding model. Most neural networks used to establish encoding models in the past were performed image classification task, the task of which is single. While in the process of human's visual perception, multiple tasks are performed simultaneously. Thus, the existing encoding model does not well satisfy the diversity and complexity of the human visual mechanism. In this paper, we first established a feature extraction model based on Fully Convolutional Network (FCN) and Visual Geometry Group (VGG) with similar network structure but different training goal, and employed Regularize Orthogonal Matching Pursuit (ROMP) to establish the response model, which can predict the stimuli-evoked responses measured by functional magnetic resonance imaging (fMRI). The results revealed that the convolutional neural networks trained by different visual tasks had significant difference in the performance of visual encoding with almost the same network structure. The VGG-based encoding model can achieve a higher performance in most voxels of ROIs. We concluded that classification task in computer vision can better fit the visual mechanism of human compared to visual segmentation task.