Semantic Correlation Mining between Images and Texts with Global Semantics and Local Mapping

This paper proposes a novel approach for the modeling of semantic correlation between web images and texts. Our approach contains two processes of semantic correlation computing. One is to find the local media objects (LMOs), the components composing text (or image) documents, that match the global semantics of a given image(or text) document based on probabilistic latent semantic analysis (PLSA); The other is to make a direct mapping among LMOs with graph-based learning, with those LMOs achieved based on PLSA as a part of inputs. The two cooperating processes consider both dominant semantics and local subordinate parts of heterogeneous data. Finally, we compute the similarity between the obtained LMOs and a whole document of the same modality and then get the semantic correlation between textual and visual documents. Experimental results demonstrate the effectiveness of the proposed approach.

[1]  Xuelong Li,et al.  Visual-Textual Joint Relevance Learning for Tag-Based Social Image Search , 2013, IEEE Transactions on Image Processing.

[2]  Xiaohua Zhai,et al.  Effective Heterogeneous Similarity Measure with Nearest Neighbors for Cross-Media Retrieval , 2012, MMM.

[3]  Tao Jiang,et al.  Learning Image-Text Associations , 2009, IEEE Transactions on Knowledge and Data Engineering.

[4]  Bingbing Ni,et al.  Assistive tagging: A survey of multimedia tagging with human-computer joint exploration , 2012, CSUR.

[5]  Lei Wu,et al.  Tag Completion for Image Retrieval , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Yi Yang,et al.  Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media Retrieval , 2008, IEEE Transactions on Multimedia.

[7]  Junping Du,et al.  Study of cross-media topic analysis based on visual topic model , 2012, 2012 24th Chinese Control and Decision Conference (CCDC).

[8]  Xiaogang Wang,et al.  Cross matching of music and image , 2012, ACM Multimedia.

[9]  Thomas Hofmann,et al.  Unsupervised Learning by Probabilistic Latent Semantic Analysis , 2004, Machine Learning.

[10]  Daniel Gatica-Perez,et al.  Modeling Semantic Aspects for Cross-Media Image Indexing , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[12]  Léon Bottou,et al.  Local Learning Algorithms , 1992, Neural Computation.

[13]  Matthieu Guillaumin,et al.  Combining Image-Level and Segment-Level Models for Automatic Annotation , 2012, MMM.

[14]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[15]  Roger Levy,et al.  A new approach to cross-modal multimedia retrieval , 2010, ACM Multimedia.