Automatic report generation based on multi-modal information

In this paper, we propose a new framework which can utilize multi-modal social media information to automatically generate related reports for users or government. First, we utilize DBSCAN (Density Based Spatial Clustering of Applications with Noise) to detect events in official news websites. Then, some unofficial information details are extracted from social network platforms (Foursquare, Twitter, YouTube), which will be leveraged to enhance the official report in order to excavate some latent and useful information. In this process, we applied some classic textual processing methods and computer vision technologies to reduce the noise information uploaded by user generated contents (UGCs). Then, we applied LSTM-CNN model to generate the related image caption and successfully convert visual information to textual information. Finally, we extracted some latent topics using graph cluster method to generate the final report. To demonstrate the effectiveness of our framework, we got a large of multi-source event dataset from official news websites and Twitter. Finally, the user study demonstrates the practicability of our approach.

[1]  Spiros Papadimitriou.  Anomaly Detection on Streams , 2009, Encyclopedia of Database Systems.

[2]  Bin Gu,et al.  Incremental Support Vector Learning for Ordinal Regression , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[3]  KhreichWael,et al.  A Survey of Techniques for Event Detection in Twitter , 2015, CI 2015.

[4]  Zhihua Xia,et al.  A Privacy-Preserving and Copy-Deterrence Content-Based Image Retrieval Scheme in Cloud Computing , 2016, IEEE Transactions on Information Forensics and Security.

[5]  Jerry L. Prince,et al.  Incompressible Deformation Estimation Algorithm (IDEA) From Tagged MR Images , 2012, IEEE Transactions on Medical Imaging.

[6]  David S. Ebert,et al.  Spatiotemporal social media analytics for abnormal event detection and examination using seasonal-trend decomposition , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[7]  Takeo Kanade,et al.  A Semi-Markov Model for Mitosis Segmentation in Time-Lapse Phase Contrast Microscopy Image Sequences of Stem Cell Populations , 2012, IEEE Transactions on Medical Imaging.

[8]  Yan-Ying Chen,et al.  Discovering the City by Mining Diverse and Multimodal Data Streams , 2014, ACM Multimedia.

[9]  Qian Wang,et al.  A Secure and Dynamic Multi-Keyword Ranked Search Scheme over Encrypted Cloud Data , 2016, IEEE Transactions on Parallel and Distributed Systems.

[10]  Ke Wang,et al.  TopicSketch: Real-Time Bursty Topic Detection from Twitter , 2013, 2013 IEEE 13th International Conference on Data Mining.

[11]  Yuting Su,et al.  Multiple/Single-View Human Action Recognition via Part-Induced Multitask Structural Learning , 2015, IEEE Transactions on Cybernetics.

[12]  Mohan S. Kankanhalli,et al.  Tweeting Cameras for Event Detection , 2015, WWW.

[13]  Hila Becker,et al.  Learning similarity metrics for event identification in social media , 2010, WSDM '10.

[14]  Bu-Sung Lee,et al.  Event Detection in Twitter , 2011, ICWSM.

[15]  Hila Becker,et al.  Identifying content for planned events across social media sites , 2012, WSDM '12.

[16]  Hanan Samet,et al.  TwitterStand: news in tweets , 2009, GIS.

[17]  Mohan S. Kankanhalli,et al.  Hierarchical Clustering Multi-Task Learning for Joint Human Action Grouping and Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Wenjie Li,et al.  Sequential Summarization: A Full View of Twitter Trending Topics , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[19]  Xiaozhong Liu,et al.  Mirroring the real world in social media: twitter, geolocation, and sentiment analysis , 2013, UnstructureNLP@CIKM.

[20]  Zhihua Xia,et al.  A Secure and Dynamic Multi-Keyword Ranked Search Scheme over Encrypted Cloud Data , 2016, IEEE Transactions on Parallel and Distributed Systems.

[21]  Naixue Xiong,et al.  EPCBIR: An efficient and privacy-preserving content-based image retrieval scheme in cloud computing , 2017, Inf. Sci..

[22]  Lexing Xie,et al.  Event Mining in Multimedia Streams , 2008, Proceedings of the IEEE.

[23]  Charu C. Aggarwal,et al.  A Survey of Text Clustering Algorithms , 2012, Mining Text Data.

[24]  James Allan,et al.  Introduction to topic detection and tracking , 2002 .

[25]  Benoit Huet,et al.  Heterogeneous features and model selection for event-based media classification , 2013, ICMR.

[26]  Qingming Huang,et al.  Cross-media topic detection: A multi-modality fusion framework , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[27]  Xiangyu Wang,et al.  Venue Semantics: Multimedia Topic Modeling of Social Media Contents , 2013, PCM.

[28]  Wael Khreich,et al.  A Survey of Techniques for Event Detection in Twitter , 2015, Comput. Intell..

[29]  Lars Schmidt-Thieme,et al.  Scalable Event-Based Clustering of Social Media Via Record Linkage Techniques , 2011, ICWSM.

[30]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[31]  Ting Wang,et al.  Information current in Twitter: which brings hot events to the world , 2013, WWW.

[32]  Liang Zhao,et al.  STED: semi-supervised targeted-interest event detectionin in twitter , 2013, KDD.

[33]  M. Huang,et al.  Value Fusion: The Blending of Consumer and Firm Value in the Distinct Context of Mobile Technologies and Social Media , 2013 .

[34]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[35]  Qi Gao,et al.  Analyzing user modeling on twitter for personalized news recommendations , 2011, UMAP'11.

[36]  Hsin-Chang Yang,et al.  A Novel Approach for Event Detection by Mining Spatio-temporal Information on Microblogs , 2011, 2011 International Conference on Advances in Social Networks Analysis and Mining.