A Study on the Analysis Model of the Ranking of the Theme of Weibo

Sina Weibo, the most popular Chinese social platform with hundreds and millions of user-contributed images and texts, is growing rapidly. However, the noise between the image and text, as well as their incomplete correspondence, makes accurate image retrieval and ranking difficult. In this paper, we propose a deep learning framework using visual features, text content and popularity of Weibo to calculate the similarity between the image and the text based on training the model to maximize the likelihood of the target description sentence given the training image. In addition, the retrieval results are reranked using the popularity of the image. The comparison experiment of the large-scale Sina Weibo dataset proves the validity of the proposed method.

[1]  Victor C. M. Leung,et al.  CAP: community activity prediction based on big data analysis , 2014, IEEE Network.

[2]  Chee Khiang Pang,et al.  An Experimental Study of Correlation between Text and Image Similarity by Information Fusion Approach , 2015 .

[3]  J. van Leeuwen,et al.  Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.

[4]  Karl Stratos,et al.  Midge: Generating Image Descriptions From Computer Vision Detections , 2012, EACL.

[5]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[6]  Ahmet Aker,et al.  Generating Image Descriptions Using Dependency Relational Patterns , 2010, ACL.

[7]  Meng Wang,et al.  Spectral Hashing With Semantically Consistent Graph for Image Indexing , 2013, IEEE Transactions on Multimedia.

[8]  Ruslan Salakhutdinov,et al.  Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models , 2014, ArXiv.

[9]  Yejin Choi,et al.  TreeTalk: Composition and Compression of Trees for Image Descriptions , 2014, TACL.

[10]  Frank Keller,et al.  Image Description using Visual Dependency Representations , 2013, EMNLP.

[11]  Zhao Wei.  Words Similarity Algorithm Based on Tongyici Cilin in Semantic Web Adaptive Learning System , 2010 .

[12]  Tao Chen,et al.  Understanding and classifying image tweets , 2013, ACM Multimedia.

[13]  Yejin Choi,et al.  Collective Generation of Natural Image Descriptions , 2012, ACL.

[14]  Gao Min.  Real-Time and Personalized Recommendation on Microblogging Systems , 2014 .

[15]  Qingshan Liu,et al.  Image retrieval via probabilistic hypergraph ranking , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Ruslan Salakhutdinov,et al.  Multimodal Neural Language Models , 2014, ICML.

[17]  Jingrui He,et al.  Generalized Manifold-Ranking-Based Image Retrieval , 2006, IEEE Transactions on Image Processing.