Multimodal Analysis of Disaster Tweets

Social media is inevitably the most abundant source of actionable information in times of natural disasters. Most of the data is either available in the form of text, images or videos. Real-time analysis of such data during the events of calamities poses many challenges to machine learning algorithms that require a large amount of data to perform well. Multimodal Twitter Dataset for Natural Disasters (CrisisMMD) is one such novel dataset that provides annotated textual as well as image data to researchers to aid the development of crisis response mechanism which can leverage social media platforms to extract useful information in times of crisis. In this paper, we analyze multimodal data related to seven different natural calamities like hurricanes, floods, earthquakes, etc. and propose a novel decision diffusion technique to classify them into informative and non-informative categories. The proposed methodology outperforms the text baselines by more than 4 % accuracy and image baselines by more than 3 %.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Rajiv Ratn Shah,et al.  Detecting Personal Intake of Medicine from Twitter , 2018, IEEE Intelligent Systems.

[3]  Muhammad Ali Ramdhani,et al.  Social Media-Based Identifier for Natural Disaster , 2018 .

[4]  Efstathios Stamatatos,et al.  Webpage Genre Identification Using Variable-Length Character n-Grams , 2007 .

[5]  Roger Zimmermann,et al.  Learning and Fusing Multimodal Deep Features for Acoustic Scene Categorization , 2018, ACM Multimedia.

[6]  Dr. med. Rajiv Shah,et al.  Multimodal Analysis of User-Generated Multimedia Content , 2017, Socio-Affective Computing.

[7]  Roger Zimmermann,et al.  Aspect-Based Financial Sentiment Analysis using Deep Learning , 2018, WWW.

[8]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[10]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Ponnurangam Kumaraguru,et al.  Mind Your Language: Abuse and Offense Detection for Code-Switched Languages , 2018, AAAI.

[12]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[13]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[14]  Rajiv Ratn Shah,et al.  Multimodal Semantics and Affective Computing from Multimedia Content , 2018 .

[15]  Yiqun Liu,et al.  Discover breaking events with popular hashtags in twitter , 2012, CIKM.

[16]  Vasudeva Varma,et al.  Deep Learning for Hate Speech Detection in Tweets , 2017, WWW.

[17]  Firoj Alam,et al.  CrisisMMD: Multimodal Twitter Datasets from Natural Disasters , 2018, ICWSM.

[18]  Nitish Srivastava,et al.  Multimodal learning with deep Boltzmann machines , 2012, J. Mach. Learn. Res..

[19]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[21]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[22]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Dinggang Shen,et al.  Deep Learning-Based Feature Representation for AD/MCI Classification , 2013, MICCAI.

[24]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[26]  Ramit Sawhney,et al.  Did you offend me? Classification of Offensive Tweets in Hinglish Language , 2018, ALW.

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  Edward Y. Chang,et al.  CBSA: content-based soft annotation for multimodal image retrieval using Bayes point machines , 2003, IEEE Trans. Circuits Syst. Video Technol..

[29]  Hermann Hellwagner,et al.  Automatic sub-event detection in emergency management using social media , 2012, WWW.

[30]  Alexander Zipf,et al.  A geographic approach for combining social media and authoritative data towards identifying useful information for disaster management , 2015, Int. J. Geogr. Inf. Sci..

[31]  Fernando Diaz,et al.  Extracting information nuggets from disaster- Related messages in social media , 2013, ISCRAM.

[32]  Andrew McCallum,et al.  A comparison of event models for naive bayes text classification , 1998, AAAI 1998.

[33]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[34]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Leon A. Gatys,et al.  Texture Synthesis Using Convolutional Neural Networks , 2015, NIPS.

[36]  Shie Mannor,et al.  Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce , 2016, AAAI 2016.

[37]  Kiyoharu Aizawa,et al.  Category-Based Deep CCA for Fine-Grained Venue Discovery From Multimodal Data , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[39]  Yi Yu,et al.  ADVISOR: Personalized Video Soundtrack Recommendation by Late Fusion with Heuristic Rankings , 2014, ACM Multimedia.

[40]  Swati Aggarwal,et al.  A Computational Approach to Feature Extraction for Identification of Suicidal Ideation in Tweets , 2018, ACL.

[41]  Ramit Sawhney,et al.  Detecting Offensive Tweets in Hindi-English Code-Switched Language , 2018, SocialNLP@ACL.

[42]  Rajiv Ratn Shah,et al.  Identification of Emergency Blood Donation Request on Twitter , 2018, EMNLP 2018.

[43]  Carolyn Penstein Rosé,et al.  Detecting offensive tweets via topical feature discovery over a large scale twitter corpus , 2012, CIKM.

[44]  Shin'ichi Satoh,et al.  Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed , 2018, ACM Multimedia.

[45]  Lei Chen,et al.  Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[46]  Ravi Shankar,et al.  A Firefly Algorithm Based Wrapper-Penalty Feature Selection Method for Cancer Diagnosis , 2018, ICCSA.

[47]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[48]  Roger Levy,et al.  A new approach to cross-modal multimedia retrieval , 2010, ACM Multimedia.