Social media and satellites

Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time. In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data. To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.

[1]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[2]  Sang Jun Park,et al.  Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks , 2017, ArXiv.

[3]  Edi Winarko,et al.  Event detection in social media: A survey , 2013, International Conference on ICT for Smart Society.

[4]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[5]  Anthony Stefanidis,et al.  #Earthquake: Twitter as a Distributed Sensor System , 2013, Trans. GIS.

[6]  Rong Du,et al.  Predicting activity attendance in event-based social networks: content, context and social influence , 2014, UbiComp.

[7]  Yannis Stavrakas,et al.  Degeneracy-Based Real-Time Sub-Event Detection in Twitter Stream , 2015, ICWSM.

[8]  Andreas Dengel,et al.  Contextual Enrichment of Remote-Sensed Events with Social Media Streams , 2016, ACM Multimedia.

[9]  Lin Li,et al.  Data-Driven Flood Detection using Neural Networks , 2017, MediaEval.

[10]  Muhammad Hanif,et al.  Flood detection using Social Media Data and Spectral Regression based Kernel Discriminant Analysis , 2017, MediaEval.

[11]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Berna Erol,et al.  Linking multimedia presentations with their symbolic source documents: algorithm and applications , 2003, ACM Multimedia.

[13]  Edson C. Tandoc,et al.  Communicating on Twitter during a disaster: An analysis of tweets during Typhoon Haiyan in the Philippines , 2015, Comput. Hum. Behav..

[14]  Andreas Dengel,et al.  Detection of Flooding Events in Social Multimedia and Satellite Imagery using Deep Neural Networks , 2017, MediaEval.

[15]  Francesco G. B. De Natale,et al.  A saliency-based approach to event recognition , 2018, Signal Process. Image Commun..

[16]  A. Fisher,et al.  Comparing Landsat water index methods for automated water classification in eastern Australia , 2016 .

[17]  Nick Koudas,et al.  TwitterMonitor: trend detection over the twitter stream , 2010, SIGMOD Conference.

[18]  Andreas Kamilaris,et al.  Disaster Monitoring using Unmanned Aerial Vehicles and Deep Learning , 2018, ArXiv.

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Yongdong Zhang,et al.  Supervised Hash Coding With Deep Neural Network for Environment Perception of Intelligent Vehicles , 2018, IEEE Transactions on Intelligent Transportation Systems.

[21]  Joost van de Weijer,et al.  Multi-modal Deep Learning Approach for Flood Detection , 2017, MediaEval.

[22]  Frank Paul,et al.  A new glacier inventory for the Svartisen region, Norway, from Landsat ETM+ data: challenges and change assessment , 2009, Journal of Glaciology.

[23]  Vijayan Sugumaran,et al.  Participatory sensing-based semantic and spatial analysis of urban emergency events using mobile social media , 2016, EURASIP J. Wirel. Commun. Netw..

[24]  Wael Khreich,et al.  A Survey of Techniques for Event Detection in Twitter , 2015, Comput. Intell..

[25]  Yongdong Zhang,et al.  Effective Uyghur Language Text Detection in Complex Background Images for Traffic Prompt Identification , 2018, IEEE Transactions on Intelligent Transportation Systems.

[26]  Martha Larson,et al.  Retrieving Social Flooding Images Based on Multimodal Information , 2017, MediaEval.

[27]  Cécile Favre,et al.  Event detection, tracking, and visualization in Twitter: a mention-anomaly-based approach , 2015, Social Network Analysis and Mining.

[28]  Michelle R. Guy,et al.  Twitter earthquake detection: earthquake monitoring in a social world , 2012 .

[29]  Marc Cheong,et al.  Twittering for Earth: A Study on the Impact of Microblogging Activism on Earth Hour 2009 in Australia , 2010, ACIIDS.

[30]  Ravindra S. Hegadi,et al.  A Survey on Multimedia Data Mining and Its Relevance Today , 2010 .

[31]  Minh-Son Dao,et al.  A Domain-based Late-Fusion for Disaster Image Retrieval from Social Media , 2017, MediaEval.

[32]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[33]  D. Guha-Sapir,et al.  EM-DAT: The CRED/OFDA International Disaster Database , 2016 .

[34]  Michael Riegler,et al.  JORD: A System for Collecting Information and Monitoring Natural Disasters by Linking Social Media with Satellite Imagery , 2017, CBMI.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Shih-Fu Chang.  New Frontiers of Large Scale Multimedia Information Retrieval , 2016, ICMR.

[37]  John Leonard Kansas,et al.  Using Landsat imagery to backcast fire and post-fire residuals in the Boreal Shield of Saskatchewan: implications for woodland caribou management , 2016 .

[38]  Soma Shiraishi,et al.  Analysis of satellite images for disaster detection , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[39]  A. F. Adams,et al.  The Survey , 2021, Dyslexia in Higher Education.

[40]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[41]  H. M. Wood,et al.  The use of Earth observing satellites for hazard support , 2001, IGARSS 2001. Scanning the Present and Resolving the Future. Proceedings. IEEE 2001 International Geoscience and Remote Sensing Symposium (Cat. No.01CH37217).

[42]  Martin Jung,et al.  Exploiting synergies of global land cover products for carbon cycle modeling , 2006 .

[43]  Michael Riegler,et al.  CNN and GAN Based Satellite and Social Media Data Fusion for Disaster Detection , 2017, MediaEval.

[44]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[45]  Michael Riegler,et al.  LIRE: open source visual information retrieval , 2016, MMSys.

[46]  Nicu Sebe,et al.  Event-based media processing and analysis: A survey of the literature , 2016, Image Vis. Comput..

[47]  Ana-Maria Popescu,et al.  Detecting controversial events from twitter , 2010, CIKM.

[48]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Steve H. L. Liang,et al.  Geo-located community detection in Twitter with enhanced fast-greedy optimization of modularity: the case study of typhoon Haiyan , 2015, Int. J. Geogr. Inf. Sci..

[50]  Jie Yin,et al.  Using Social Media to Enhance Emergency Situation Awareness , 2012, IEEE Intelligent Systems.

[51]  Sergey V. Samsonov,et al.  A review of the status of satellite remote sensing and image processing techniques for mapping natural hazards and disasters , 2009 .

[52]  Ying Liu,et al.  Geological Disaster Recognition on Optical Remote Sensing Images Using Deep Learning , 2016 .

[53]  Chenliang Li,et al.  Twevent: segment-based event detection from tweets , 2012, CIKM.

[54]  Arkaitz Zubiaga,et al.  WISC at MediaEval 2017: Multimedia Satellite Task , 2017, MediaEval.

[55]  Michael Riegler,et al.  Efficient disease detection in gastrointestinal videos – global features versus neural networks , 2017, Multimedia Tools and Applications.

[56]  Yiannis Kompatsiaris,et al.  Visual and Textual Analysis of Social Media and Satellite Images for Flood Detection @ Multimedia Satellite Task MediaEval 2017 , 2017, MediaEval.

[57]  Nicola Conci,et al.  Convolutional Neural Networks for Disaster Images Retrieval , 2017, MediaEval.

[58]  Andreas Dengel,et al.  The Multimedia Satellite Task at MediaEval 2018 , 2017, MediaEval.

[59]  J. Campbell.  Introduction to remote sensing , 1987 .

[60]  Tao Chen,et al.  DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks , 2014, ArXiv.

[61]  Michael Riegler,et al.  KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection , 2017, MMSys.

[62]  Francesco G. B. De Natale,et al.  A hierarchical approach to event discovery from single images using MIL framework , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[63]  Martha Larson,et al.  Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks , 2016, 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI).