Heterogeneous multimedia cooperative annotation based on multimodal correlation learning

Abstract Rich multimedia contents are dominating the current Web. In popular social media platforms such as FaceBook, Twitter, and Instagram, there are over millions of multimedia contents being created by users. In the meantime, multimedia data consists of data in multiple modalities, such as text, images, videos, audio, time series sequences, and so on. Many research efforts have been devoted to multimedia annotation to further improve the performance. However, the prevailing methods are designed for single-media annotation task. In fact, heterogeneous media content describes given labels from respective modality and is complementary to each other, and it becomes critical to explore advanced techniques for heterogeneous data analysis and multimedia annotation. Inspired by this idea, this paper presents a new multimodal correlation learning method for heterogeneous multimedia cooperative annotation, named unified space learning, which projects heterogeneous media data into one unified space. We formulate the multimedia annotation task into a semi-supervised learning framework, in which we learn different projection matrices for different media type. By doing so, different media content is aligned cooperatively, and jointly provides a more complementary profile of given semantic labels. Experimental results on data set with images, audio clips, videos and 3D models show that the proposed approach is more effective.