Multimodal topic modeling based geo-annotation for social event detection in large photo collections

Multimodal image clustering becomes an effective approach for social event detection in large photo collections. In addition to visual and textual information, geographic information can also be used to improve the detection accuracy of social events. However, not every image in a photo collection is tagged with geographic information. A topic model based approach is proposed to estimate missing geographic information in a photo which involves a supervised multimodal model to estimated the joint distribution of time, geographic, content, and textual information for a large set of photos. The photos without geographic information are annotated with a predicted geographic coordinate. We show the efficacy of the proposed approach for event detection and annotation from a large photo collection.

[1]  Ron Bekkerman,et al.  Multi-modal Clustering for Multimedia Collections , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Michael I. Jordan,et al.  Modeling annotated data , 2003, SIGIR.

[3]  Yiannis Kompatsiaris,et al.  Social event detection using multimodal clustering and integrating supervisory signals , 2012, ICMR.

[4]  Jiebo Luo,et al.  Geotagging in multimedia and computer vision—a survey , 2010, Multimedia Tools and Applications.

[5]  Yiannis Kompatsiaris,et al.  Collaborative event annotation in tagged photo collections , 2012, Multimedia Tools and Applications.

[6]  Yiannis Kompatsiaris,et al.  Social Event Detection at MediaEval 2012: Challenges, Dataset and Evaluation , 2012, MediaEval.

[7]  Hugo Larochelle,et al.  Topic Modeling of Multimodal Data: An Autoregressive Approach , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Yiannis Kompatsiaris,et al.  Graph-Based Multimodal Clustering for Social Event Detection in Large Collections of Images , 2014, MMM.

[9]  Richi Nayak,et al.  The Ranking Based Constrained Document Clustering Method and Its Application to Social Event Detection , 2014, DASFAA.

[10]  Hila Becker,et al.  Learning similarity metrics for event identification in social media , 2010, WSDM '10.

[11]  Philipp Cimiano,et al.  Event-based classification of social media streams , 2012, ICMR.

[12]  Durga Toshniwal,et al.  Sub-event detection during natural hazards using features of social media data , 2013, WWW.

[13]  Hermann Hellwagner,et al.  Social media for crisis management: clustering approaches for sub-event detection , 2015, Multimedia Tools and Applications.

[14]  Hugo Larochelle,et al.  A Neural Autoregressive Topic Model , 2012, NIPS.

[15]  R. Mooney,et al.  Impact of Similarity Measures on Web-page Clustering , 2000 .

[16]  Lars Schmidt-Thieme,et al.  Scalable Event-Based Clustering of Social Media Via Record Linkage Techniques , 2011, ICWSM.

[17]  Thomas Hofmann,et al.  Probabilistic Latent Semantic Analysis , 1999, UAI.

[18]  Touradj Ebrahimi,et al.  Geotag propagation in social networks based on user trust model , 2010, Multimedia Tools and Applications.