A deep multi-modal neural network for informative Twitter content classification during emergencies

People start posting tweets containing texts, images, and videos as soon as a disaster hits an area. The analysis of these disaster-related tweet texts, images, and videos can help humanitarian response organizations in better decision-making and prioritizing their tasks. Finding the informative contents which can help in decision making out of the massive volume of Twitter content is a difficult task and require a system to filter out the informative contents. In this paper, we present a multi-modal approach to identify disaster-related informative content from the Twitter streams using text and images together. Our approach is based on long-short-term-memory and VGG-16 networks that show significant improvement in the performance, as evident from the validation result on seven different disaster-related datasets. The range of F1-score varied from 0.74 to 0.93 when tweet texts and images used together, whereas, in the case of only tweet text, it varies from 0.61 to 0.92. From this result, it is evident that the proposed multi-modal system is performing significantly well in identifying disaster-related informative social media contents.

[1]  Muhammad Imran,et al.  Damage Assessment from Social Media Imagery Data During Disasters , 2017, 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[2]  Anupam Joshi,et al.  Faking Sandy: characterizing and identifying fake images on Twitter during Hurricane Sandy , 2013, WWW.

[3]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[4]  Yogesh Kumar Dwivedi,et al.  Event classification and location prediction from tweets during disasters , 2017, Annals of Operations Research.

[5]  Carlos Castillo,et al.  AIDR: artificial intelligence for disaster response , 2014, WWW.

[6]  Shahriar Akter,et al.  Big data and disaster management: a systematic review and agenda for future research , 2017, Annals of Operations Research.

[7]  Abhinav Kumar,et al.  Location reference identification from tweets during emergencies: A deep learning approach , 2019, International Journal of Disaster Risk Reduction.

[8]  Fernando Diaz,et al.  CrisisLex: A Lexicon for Collecting and Filtering Microblogged Communications in Crises , 2014, ICWSM.

[9]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[10]  Makarand Hastak,et al.  Social network analysis: Characteristics of online social networks after a disaster , 2018, Int. J. Inf. Manag..

[11]  Zongwei Luo,et al.  Developing an Integration Framework for Crowdsourcing and Internet of Things with Applications for Disaster Response , 2015, 2015 IEEE International Conference on Data Science and Data Intensive Systems.

[12]  Leysia Palen,et al.  Natural Language Processing to the Rescue? Extracting "Situational Awareness" Tweets During Mass Emergency , 2011, ICWSM.

[13]  Mariette Awad,et al.  A computationally efficient multi-modal classification approach of disaster-related Twitter images , 2019, SAC.

[14]  Qunying Huang,et al.  Geographic Situational Awareness: Mining Tweets for Disaster Preparedness, Emergency Response, Impact, and Recovery , 2015, ISPRS Int. J. Geo Inf..

[15]  Shady Elbassuoni,et al.  Practical extraction of disaster-relevant information from social media , 2013, WWW.

[16]  Yutaka Matsuo,et al.  Tweet Analysis for Real-Time Event Detection and Earthquake Reporting System Development , 2013, IEEE Transactions on Knowledge and Data Engineering.

[17]  Ana Beatriz Lopes de Sousa Jabbour,et al.  An analysis of the literature on humanitarian logistics and supply chain management: paving the way for future studies , 2019, Ann. Oper. Res..

[18]  Mariette Awad,et al.  Damage Identification in Social Media Posts using Multimodal Deep Learning , 2018, ISCRAM.

[19]  Yogesh Kumar Dwivedi.  Social media marketing and advertising , 2015 .

[20]  Yogesh Kumar Dwivedi,et al.  Social media in marketing: A review and analysis of the existing literature , 2017, Telematics Informatics.

[21]  Yogesh Kumar Dwivedi,et al.  Involvement in emergency supply chain for disaster management: a cognitive dissonance perspective , 2018, Int. J. Prod. Res..

[22]  Makarand Hastak,et al.  Emergency information diffusion on online social media during storm Cindy in U.S , 2018, Int. J. Inf. Manag..

[23]  Deborah Bunker,et al.  Emergency management in the changing world of social media: Framing the research agenda with the stakeholders through engaged scholarship , 2019, Int. J. Inf. Manag..

[24]  Asif Ekbal,et al.  Linguistic Feature Assisted Deep Learning Approach towards Multi-label Classification of Crisis Related Tweets , 2018, ISCRAM.

[25]  Padmanabha Aital,et al.  Mechanics of humanitarian supply chain agility and resilience and its empirical validation , 2014 .

[26]  A. Gunasekaran,et al.  The role of Big Data in explaining disaster resilience in supply chains for sustainability , 2017 .

[27]  Yogesh Kumar Dwivedi,et al.  Disaster management in Bangladesh: developing an effective emergency supply chain network , 2018, Ann. Oper. Res..

[28]  Aron Culotta,et al.  Tweedr: Mining twitter to inform disaster response , 2014, ISCRAM.

[29]  Firoj Alam,et al.  Image4Act: Online Social Media Image Processing for Disaster Response , 2017, 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[30]  Kyungsup Kim,et al.  A logistics model for the transport of disaster victims with various injuries and survival probabilities , 2015, Ann. Oper. Res..

[31]  Yogesh Kumar Dwivedi,et al.  Impact of internet of things (IoT) in disaster management: a task-technology fit perspective , 2019, Ann. Oper. Res..

[32]  Wael Khreich,et al.  A Survey of Techniques for Event Detection in Twitter , 2015, Comput. Intell..

[33]  Jintae Lee,et al.  Content features of tweets for effective communication during disasters: A media synchronicity theory perspective , 2019, Int. J. Inf. Manag..

[34]  Werner Retschitzegger,et al.  Cross-domain informativeness classification for disaster situations , 2018, MEDES.

[35]  Jie Yin,et al.  Emergency situation awareness from twitter for crisis management , 2012, WWW.

[36]  Yogesh Kumar Dwivedi,et al.  Advances in Social Media Research: Past, Present and Future , 2017, Information Systems Frontiers.

[37]  Aixin Sun,et al.  A Survey of Location Prediction on Twitter , 2017, IEEE Transactions on Knowledge and Data Engineering.

[38]  Hassan Sajjad,et al.  Rapid Classification of Crisis-Related Data on Social Networks using Convolutional Neural Networks , 2016, ICWSM 2016.

[39]  Vidhyacharan Bhaskar,et al.  Big data analytics for disaster response and recovery through sentiment analysis , 2018, Int. J. Inf. Manag..

[40]  Jyoti Prakash Singh,et al.  A Comparative Analysis of Machine Learning Techniques for Disaster-Related Tweet Classification , 2019, 2019 IEEE R10 Humanitarian Technology Conference (R10-HTC)(47129).

[41]  KimJooho,et al.  Social Network Analysis , 2018 .

[42]  Jomon Aliyas Paul,et al.  Location-allocation planning of stockpiles for effective disaster mitigation , 2012, Annals of Operations Research.

[43]  Gunjan Soni,et al.  Modelling the inter-relationship between factors affecting coordination in a humanitarian supply chain: a case of Chennai flood relief , 2018, Ann. Oper. Res..

[44]  Muhammad Imran,et al.  Summarizing Situational Tweets in Crisis Scenario , 2016, HT.

[45]  Rameshwar Dubey,et al.  Swift trust and commitment: The missing links for humanitarian supply chain coordination? , 2019, Ann. Oper. Res..

[46]  Guofeng Cao,et al.  Social media data and post-disaster recovery , 2019, Int. J. Inf. Manag..

[47]  KhreichWael,et al.  A Survey of Techniques for Event Detection in Twitter , 2015, CI 2015.

[48]  John Yen,et al.  Classifying text messages for the haiti earthquake , 2011, ISCRAM.

[49]  Sarah Vieweg,et al.  Processing Social Media Messages in Mass Emergency , 2014, ACM Comput. Surv..

[50]  Tomasz Bednarz,et al.  Image Classification to Support Emergency Situation Awareness , 2016, Front. Robot. AI.

[51]  Firoj Alam,et al.  CrisisMMD: Multimodal Twitter Datasets from Natural Disasters , 2018, ICWSM.

[52]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[53]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[54]  Cornelia Caragea,et al.  Identifying informative messages in disaster events using Convolutional Neural Networks , 2016 .

[55]  Abhinav Kumar,et al.  Authenticity of Geo-Location and Place Name in Tweets , 2017, AMCIS.

[56]  Fernando Diaz,et al.  Extracting information nuggets from disaster- Related messages in social media , 2013, ISCRAM.

[57]  James A. Thom,et al.  Mining and Classifying Image Posts on Social Media to Analyse Fires , 2016, ISCRAM.

[58]  Wenlin Yao,et al.  Social media for intelligent public information and warning in disasters: An interdisciplinary review , 2019, Int. J. Inf. Manag..

[59]  Indranil Bose,et al.  Application of Image Analytics for Disaster Response in Smart Cities , 2019, HICSS.

[60]  Qunying Huang,et al.  Deep learning for real-time social media text classification for situation awareness – using Hurricanes Sandy, Harvey, and Irma as case studies , 2019, Int. J. Digit. Earth.

[61]  Cyril R. H. Foropon,et al.  Big data analytics and organizational culture as complements to swift trust and collaborative performance in the humanitarian supply chain , 2019, International Journal of Production Economics.

[62]  Cornelia Caragea,et al.  Twitter Mining for Disaster Response: A Domain Adaptation Approach , 2015, ISCRAM.