Tracking Natural Events through Social Media and Computer Vision

Accurate, efficient, global observation of natural events is important for ecologists, meteorologists, governments, and the public. Satellites are effective but limited by their perspective and by atmospheric conditions. Public images on photo-sharing websites could provide crowd-sourced ground data to complement satellites, since photos contain evidence of the state of the natural world. In this work, we test the ability of computer vision to observe natural events in millions of geo-tagged Flickr photos, over nine years and an entire continent. We use satellites as (noisy) ground truth to train two types of classifiers, one that estimates if a Flickr photo has evidence of an event, and one that aggregates these estimates to produce an observation for given times and places. We present a web tool for visualizing the satellite and photo observations, allowing scientists to explore this novel combination of data sources.

[1]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  References , 1971 .

[3]  Marco Tagliasacchi,et al.  Snow phenomena modeling through online public media , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[4]  Laura Schweitzer,et al.  Advances In Kernel Methods Support Vector Learning , 2016 .

[5]  Shawn D. Newsam,et al.  Proximate sensing: Inferring what-is-where from georeferenced photo collections , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  A. Guerry,et al.  Using social media to quantify nature-based tourism and recreation , 2013, Scientific Reports.

[7]  D. Lazer,et al.  The Parable of Google Flu: Traps in Big Data Analysis , 2014, Science.

[8]  Robert Pless,et al.  Webcam2Satellite: Estimating cloud maps from webcam imagery , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[9]  Jean-Jacques Bedet,et al.  Distributed Active Archive Center , 1993 .

[10]  Charu C. Aggarwal,et al.  Social Sensing , 2013, Managing and Mining Sensor Data.

[11]  Johan Bollen,et al.  Twitter mood predicts the stock market , 2010, J. Comput. Sci..

[12]  Jon M. Kleinberg,et al.  Mapping the world's photos , 2009, WWW '09.

[13]  Antonio Torralba,et al.  Anticipating Visual Representations from Unlabeled Video , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Xiaofeng Tao,et al.  Transient attributes for high-level understanding and editing of outdoor scenes , 2014, ACM Trans. Graph..

[15]  Robert Pless,et al.  Building Dynamic Cloud Maps from the Ground Up , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  Bolei Zhou,et al.  Recognizing City Identity via Attribute Analysis of Geo-tagged Images , 2014, ECCV.

[17]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[18]  David J. Crandall,et al.  Where have all the people gone? Enhancing global conservation using night lights and social media. , 2015, Ecological applications : a publication of the Ecological Society of America.

[19]  Stefan Lee,et al.  Predicting Geo-informative Attributes in Large-Scale Image Collections Using Convolutional Neural Networks , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[20]  Marco Tagliasacchi,et al.  SnowWatch: Snow Monitoring through Acquisition and Analysis of User-Generated Content , 2015, ArXiv.

[21]  Jiebo Luo,et al.  Using user generated online photos to estimate and monitor air pollution in major cities , 2015, ICIMCS '15.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  David J. Crandall,et al.  Observing the Natural World with Flickr , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[24]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[25]  Milan Šećerov,et al.  Analysis of panoramio photo tags in order to extract land use information , 2015 .

[26]  David J. Crandall,et al.  Mining photo-sharing websites to study ecological phenomena , 2012, WWW.

[27]  Pascal Fua,et al.  Hot or Not: Exploring Correlations between Appearance and Temperature , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Jiebo Luo,et al.  A Multifaceted Approach to Social Multimedia-Based Prediction of Elections , 2015, IEEE Transactions on Multimedia.

[29]  Jeremy Ginsberg,et al.  Detecting influenza epidemics using search engine query data , 2009, Nature.