Deep Semantic Mapping for Cross-Modal Retrieval

Cross-Modal mapping plays an essential role in multimedia information retrieval systems. However, most of existing work paid much attention on learning mapping functions but neglected the exploration of high-level semantic representation of modalities. Inspired by recent success of deep learning, in this paper, deep CNN (convolutional neural networks) features and topic features are utilized as visual and textual semantic representation respectively. To investigate the highly non-linear semantic correlation between image and text, we propose a regularized deep neural network(RE-DNN) for semantic mapping across modalities. By imposing intra-modal regularization as supervised pre-training, we finally learn a joint model which captures both intra-modal and inter-modal relationships. Our approach is superior to previous work in follows: (1) it explores high-level semantic correlations, (2) it requires little prior knowledge for model training, (3) it is able to tackle modality missing problem. Extensive experiments on benchmark Wikipedia dataset show RE-DNN outperforms the state-of-the-art approaches in cross-modal retrieval.