Contextual Enrichment of Remote-Sensed Events with Social Media Streams

The availability of satellite images for academic or commercial purpose is increasing rapidly due to efforts made by governmental agencies (NASA, ESA) to publish such data openly or commercial startups (PlanetLabs) to provide real-time satellite data. Beyond many commercial application, satellite data is helpful to create situation awareness in disaster recovery and emergency situations such as wildfires, earthquakes, or flooding. To fully utilize such data sources, we present a scalable system for the contextual enrichment of satellite images by crawling and analyzing multimedia content from social media. This information stream can provide vital information from the ground and help to complement remote sensing in situations. We use Twitter as main data source and analyze its textual, visual, temporal, geographical and social dimensions. Visualizations show different aspects of the event allowing high-level comprehension and provide deeper insights into the event as complemented by social media.

[1]  Joemon M. Jose,et al.  "Picture the scene...";: Visually Summarising Social Media Events , 2014, CIKM.

[2]  J. Sarasohn-Kahn.  The Wisdom of Patients: Health Care Meets Online Social Media , 2008 .

[3]  David A. Shamma,et al.  YFCC100M , 2015, Commun. ACM.

[4]  Adrian Ulges,et al.  Pornography detection in video benefits (a lot) from a multi-modal approach , 2012, AMVA '12.

[5]  E. Noji,et al.  (A335) Emergent use of Social Media: A New Age of Opportunity for Disaster Resilience , 2011, Prehospital and Disaster Medicine.

[6]  A. Pentland,et al.  Life in the network: The coming age of computational social science: Science , 2009 .

[7]  Adrian Ulges,et al.  Visual Concept Learning from Weakly Labeled Web Videos , 2010, Video Search and Mining.

[8]  D. Maynard,et al.  Challenges in developing opinion mining tools for social media , 2012 .

[9]  C. Schmidt.  Trending Now: Using Social Media to Predict and Track Disease Outbreaks , 2012, Environmental health perspectives.

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Wu He,et al.  International Journal of Information Management Social Media Competitive Analysis and Text Mining: a Case Study in the Pizza Industry , 2022 .

[12]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[13]  Jie Yin,et al.  Using Social Media to Enhance Emergency Situation Awareness , 2012, IEEE Intelligent Systems.

[14]  L. Palen.  Online Social Media in Crisis Events. , 2008 .

[15]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Hila Becker,et al.  Identifying content for planned events across social media sites , 2012, WSDM '12.

[17]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[18]  ThelwallMike,et al.  Sentiment strength detection in short informal text , 2010 .

[19]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[20]  Tao Chen,et al.  DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks , 2014, ArXiv.

[21]  Cordelia Schmid,et al.  Evaluation of GIST descriptors for web-scale image search , 2009, CIVR '09.

[22]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[23]  B. Lindsay.  Social Media and Disasters: Current Uses, Future Options, and Policy Considerations , 2011 .

[24]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[25]  Guillaume Bouchard,et al.  Opinion mining in social media: Modeling, simulating, and forecasting political opinions in the web , 2012, Gov. Inf. Q..