A Comparative Study of Global and Deep Features for the Analysis of User-Generated Natural Disaster Related Images

The paper addresses the problem of adverse events (natural disasters) recognition in user-generated images from social media, addressing the problem from two complementary perspectives. On one side, we aim to provide a comprehensive comparative analysis of different feature extraction and classification algorithms, relying on two different families of feature extraction algorithms, namely (i) Global features and (ii) Deep features. On the other hand, we demonstrate that the fusion of different feature extraction and classification strategies can outperform the single methods by jointly exploiting the capabilities of individual feature descriptors. The evaluation of the methods are carried out on two datasets, including a benchmark and a self-collected dataset.

[1]  Minh-Son Dao,et al.  A Domain-based Late-Fusion for Disaster Image Retrieval from Social Media , 2017, MediaEval.

[2]  Pablo N. Mendes,et al.  Improving efficiency and accuracy in multilingual entity extraction , 2013, I-SEMANTICS '13.

[3]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[4]  Lin Li,et al.  Data-Driven Flood Detection using Neural Networks , 2017, MediaEval.

[5]  Andreas Dengel,et al.  Detection of Flooding Events in Social Multimedia and Satellite Imagery using Deep Neural Networks , 2017, MediaEval.

[6]  Jie Yin,et al.  Using Social Media to Enhance Emergency Situation Awareness , 2012, IEEE Intelligent Systems.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Michael Riegler,et al.  CNN and GAN Based Satellite and Social Media Data Fusion for Disaster Detection , 2017, MediaEval.

[9]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  David A. Shamma,et al.  YFCC100M , 2015, Commun. ACM.

[11]  Michael Riegler,et al.  The JORD System: Linking Sky and Social Multimedia Data to Natural Disasters , 2017, ICMR.

[12]  Michael Riegler,et al.  LIRE: open source visual information retrieval , 2016, MMSys.

[13]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Tao Chen,et al.  DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks , 2014, ArXiv.

[15]  Shih-Fu Chang,et al.  Deep Cross Residual Learning for Multitask Visual Recognition , 2016, ACM Multimedia.

[16]  Nicola Conci,et al.  Convolutional Neural Networks for Disaster Images Retrieval , 2017, MediaEval.

[17]  Francesco G. B. De Natale,et al.  Event recognition in personal photo collections via multiple instance learning-based classification of multiple images , 2017, J. Electronic Imaging.

[18]  Muhammad Hanif,et al.  Flood detection using Social Media Data and Spectral Regression based Kernel Discriminant Analysis , 2017, MediaEval.

[19]  Andreas Dengel,et al.  The Multimedia Satellite Task at MediaEval 2018 , 2017, MediaEval.

[20]  Michael Riegler,et al.  JORD: A System for Collecting Information and Monitoring Natural Disasters by Linking Social Media with Satellite Imagery , 2017, CBMI.

[21]  Francesco G. B. De Natale,et al.  USED: a large-scale social event detection dataset , 2016, MMSys.

[22]  Francesco G. B. De Natale,et al.  A saliency-based approach to event recognition , 2018, Signal Process. Image Commun..

[23]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Martha Larson,et al.  Retrieving Social Flooding Images Based on Multimodal Information , 2017, MediaEval.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Nicu Sebe,et al.  Event-based media processing and analysis: A survey of the literature , 2016, Image Vis. Comput..

[27]  Yiannis Kompatsiaris,et al.  Visual and Textual Analysis of Social Media and Satellite Images for Flood Detection @ Multimedia Satellite Task MediaEval 2017 , 2017, MediaEval.

[28]  Joost van de Weijer,et al.  Multi-modal Deep Learning Approach for Flood Detection , 2017, MediaEval.

[29]  Farid Melgani,et al.  A pool of deep models for event recognition , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[30]  Michael Riegler,et al.  A Comparison of Deep Learning with Global Features for Gastrointestinal Disease Detection , 2017, MediaEval.

[31]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[32]  Arkaitz Zubiaga,et al.  WISC at MediaEval 2017: Multimedia Satellite Task , 2017, MediaEval.