Automatic detection of passable roads after floods in remote sensed and social media data

This paper addresses the problem of floods classification and floods aftermath detection utilizing both social media and satellite imagery. Automatic detection of disasters such as floods is still a very challenging task. The focus lies on identifying passable routes or roads during floods. Two novel solutions are presented, which were developed for two corresponding tasks at the MediaEval 2018 benchmarking challenge. The tasks are (i) identification of images providing evidence for road passability and (ii) differentiation and detection of passable and non-passable roads in images from two complementary sources of information. For the first challenge, we mainly rely on object and scene-level features extracted through multiple deep models pre-trained on the ImageNet and Places datasets. The object and scene-level features are then combined using early, late and double fusion techniques. To identify whether or not it is possible for a vehicle to pass a road in satellite images, we rely on Convolutional Neural Networks and a transfer learning-based classification approach. The evaluation of the proposed methods are carried out on the large-scale datasets provided for the benchmark competition. The results demonstrate significant improvement in the performance over the recent state-of-art approaches.

[1]  Michael Riegler,et al.  CNN and GAN Based Satellite and Social Media Data Fusion for Disaster Detection , 2017, MediaEval.

[2]  Yiannis S. Boutalis,et al.  CEDD: Color and Edge Directivity Descriptor: A Compact Descriptor for Image Indexing and Retrieval , 2008, ICVS.

[3]  David A. Shamma,et al.  YFCC100M , 2015, Commun. ACM.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[6]  Abdullah Bulbul,et al.  Social media based 3D visual popularity , 2017, Comput. Graph..

[7]  Martha Larson,et al.  Exploiting Local Semantic Concepts for Flooding-related Social Image Classification , 2018, MediaEval.

[8]  Yiannis Kompatsiaris,et al.  Visual and Textual Analysis of Social Media and Satellite Images for Flood Detection @ Multimedia Satellite Task MediaEval 2017 , 2017, MediaEval.

[9]  Minh-Son Dao,et al.  A Domain-based Late-Fusion for Disaster Image Retrieval from Social Media , 2017, MediaEval.

[10]  Lianwen Jin,et al.  A novel feature extraction method using Pyramid Histogram of Orientation Gradients for smile recognition , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[11]  Richard Szeliski,et al.  Modeling the World from Internet Photo Collections , 2008, International Journal of Computer Vision.

[12]  Muhammad Hanif,et al.  Flood detection using Social Media Data and Spectral Regression based Kernel Discriminant Analysis , 2017, MediaEval.

[13]  Farid Melgani,et al.  Ensemble of Deep Models for Event Recognition , 2018, ACM Trans. Multim. Comput. Commun. Appl..

[14]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[16]  Abdullah Bulbul,et al.  Populating virtual cities using social media , 2017, Comput. Animat. Virtual Worlds.

[17]  Michael Riegler,et al.  JORD: A System for Collecting Information and Monitoring Natural Disasters by Linking Social Media with Satellite Imagery , 2017, CBMI.

[18]  Akio Yamada,et al.  The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[19]  J. S. Verkade,et al.  Probabilistic flood extent estimates from social media flood observations , 2016 .

[20]  Rozenn Dahyot,et al.  Automatic Discovery and Geotagging of Objects from Street View Imagery , 2017, Remote. Sens..

[21]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[22]  Shu-Ching Chen,et al.  Hierarchical disaster image classification for situation report enhancement , 2011, 2011 IEEE International Conference on Information Reuse & Integration.

[23]  Michael Riegler,et al.  Deep Learning and Hand-Crafted Feature Based Approaches for Polyp Detection in Medical Videos , 2018, 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS).

[24]  Ying Liu,et al.  Geological Disaster Recognition on Optical Remote Sensing Images Using Deep Learning , 2016 .

[25]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[26]  Markus Seidl,et al.  Detection of Road Passability from Social Media and Satellite Images , 2018, MediaEval.

[27]  D. Massart,et al.  The Mahalanobis distance , 2000 .

[28]  Francesco G. B. De Natale,et al.  A Comparative Study of Global and Deep Features for the Analysis of User-Generated Natural Disaster Related Images , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  V. Klemas,et al.  Remote Sensing of Floods and Flood-Prone Areas: An Overview , 2015 .

[31]  John Leonard Kansas,et al.  Using Landsat imagery to backcast fire and post-fire residuals in the Boreal Shield of Saskatchewan: implications for woodland caribou management , 2016 .

[32]  Soma Shiraishi,et al.  Analysis of satellite images for disaster detection , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[33]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[35]  Georgios Meditskos,et al.  A Multimodal Approach in Estimating Road Passability Through a Flooded Area Using Social Media and Satellite Images , 2018, MediaEval.

[36]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[37]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[38]  Paolo Garza,et al.  Deep Learning Models for Passability Detection of Flooded Roads , 2018, MediaEval.

[39]  Michael Riegler,et al.  A Holistic Multimedia System for Gastrointestinal Tract Disease Detection , 2017, MMSys.

[40]  Arkaitz Zubiaga,et al.  WISC at MediaEval 2017: Multimedia Satellite Task , 2017, MediaEval.

[41]  Carlos Castillo,et al.  AIDR: artificial intelligence for disaster response , 2014, WWW.

[42]  Andreas Dengel,et al.  Detection of Flooding Events in Social Multimedia and Satellite Imagery using Deep Neural Networks , 2017, MediaEval.

[43]  Tomasz Bednarz,et al.  Image Classification to Support Emergency Situation Awareness , 2016, Front. Robot. AI.

[44]  Michael Riegler,et al.  Deep learning and handcrafted feature based approaches for automatic detection of angiectasia , 2018, 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[45]  Sang Jun Park,et al.  Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks , 2017, ArXiv.

[46]  Michael Riegler,et al.  Social media and satellites , 2019, Multimedia Tools and Applications.

[47]  Nicola Conci,et al.  Convolutional Neural Networks for Disaster Images Retrieval , 2017, MediaEval.

[48]  Andreas Dengel,et al.  The Multimedia Satellite Task at MediaEval 2018 , 2017, MediaEval.

[49]  Michael Riegler,et al.  The JORD System: Linking Sky and Social Multimedia Data to Natural Disasters , 2017, ICMR.

[50]  Muhammad Hanif,et al.  Detection of Passable Roads Using Ensemble of Global and Local Features , 2018, MediaEval.

[51]  Firoj Alam,et al.  Processing Social Media Images by Combining Human and Machine Computing during Crises , 2018, Int. J. Hum. Comput. Interact..

[52]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Otávio A. B. Penatti,et al.  Exploiting ConvNet Diversity for Flooding Identification , 2017, IEEE Geoscience and Remote Sensing Letters.

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Bryan Scotney,et al.  Flood Event Image Recognition via Social Media Image and Text Analysis , 2016 .

[56]  Monika Sester,et al.  Ensembled Convolutional Neural Network Models for Retrieving Flood Relevant Tweets , 2018, MediaEval.

[57]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[58]  Jie Yin,et al.  Using Social Media to Enhance Emergency Situation Awareness , 2012, IEEE Intelligent Systems.