Predicting Livelihood Indicators from Community-Generated Street-Level Imagery

Major decisions from governments and other large organizations rely on measurements of the populace’s well-being, but making such measurements at a broad scale is expensive and thus infrequent in much of the developing world. We propose an inexpensive, scalable, and interpretable approach to predict key livelihood indicators from public crowd-sourced street-level imagery. Such imagery can be cheaply collected and more frequently updated compared to traditional surveying methods, while containing plausibly relevant information for a range of livelihood indicators. We propose two approaches to learn from the street-level imagery: (1) a method that creates multi-household cluster representations by detecting informative objects and (2) a graph-based approach that captures the relationships between images. By visualizing what features are important to a model and how they are used, we can help end-user organizations understand the models and offer an alternate approach for index estimation that uses cheaply obtained roadway features. By comparing our results against ground data collected in nationallyrepresentative household surveys, we demonstrate the performance of our approach in accurately predicting indicators of poverty, population, and health and its scalability by testing in two different countries, India and Kenya. Our code is available at https://github.com/sustainlab-group/mapillarygcn.

[1]  Anne Driscoll,et al.  Using publicly available satellite imagery and deep learning to understand economic well-being in Africa , 2020, Nature Communications.

[2]  Stefano Ermon,et al.  Learning to Interpret Satellite Images in Global Scale Using Wikipedia , 2019, ArXiv.

[3]  Virgilio Galdo,et al.  Identifying Urban Areas by Combining Human Judgment and Machine Learning: An Application to India , 2019, Journal of Urban Economics.

[4]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jiale Wang,et al.  A human-machine adversarial scoring framework for urban perception assessment using street-view images , 2019, Int. J. Geogr. Inf. Sci..

[6]  Jay Taneja,et al.  Street smarts: measuring intercity road quality using deep learning on satellite imagery , 2019, COMPASS.

[7]  Marco Aiello,et al.  AAAI Conference on Artificial Intelligence , 2011, AAAI Conference on Artificial Intelligence.

[8]  Chandan Singh,et al.  Definitions, methods, and applications in interpretable machine learning , 2019, Proceedings of the National Academy of Sciences.

[9]  Jonathan Krause,et al.  Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States , 2017, Proceedings of the National Academy of Sciences.

[10]  Achim Zeileis,et al.  Bias in random forest variable importance measures: Illustrations, sources and a solution , 2007, BMC Bioinformatics.

[11]  M. Ezzati,et al.  Measuring social, environmental and health inequalities using deep learning and street imagery , 2019, Scientific Reports.

[12]  Ramesh Raskar,et al.  Streetscore -- Predicting the Perceived Safety of One Million Streetscapes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[13]  S. Ermon,et al.  Generating Interpretable Poverty Maps using Object Detection in Satellite Images , 2020, IJCAI.

[14]  UN GLOBAL PULSE.  Mining Indonesian Tweets to Understand Food Price Crises , 2014 .

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Stefano Ermon,et al.  Efficient Poverty Mapping using Deep Reinforcement Learning , 2020, ArXiv.

[17]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[18]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Andrew Head,et al.  Can Human Development be Measured with Satellite Imagery? , 2017, ICTD.

[20]  Lorenzo Porzi,et al.  Seamless Scene Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Zhaoya Gong,et al.  Classifying Street Spaces with Street View Images for a Spatial Indicator of Urban Functions , 2019, Sustainability.

[22]  Chang-Tien Lu,et al.  StreetNet: preference learning with convolutional neural network on urban crime perception , 2018, SIGSPATIAL/GIS.

[23]  Sang Michael Xie,et al.  Combining satellite imagery and machine learning to predict poverty , 2016, Science.

[24]  Claudia Hauff,et al.  A study on the accuracy of Flickr's geotag data , 2013, SIGIR.

[25]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[26]  Roland Hodler,et al.  Nighttime lights as a proxy for human development at the local level , 2017, PloS one.

[27]  Gabriel Cadamuro,et al.  Predicting poverty and wealth from mobile phone metadata , 2015, Science.

[28]  Alberto Maria Segre,et al.  The Use of Twitter to Track Levels of Disease Activity and Public Concern in the U.S. during the Influenza A H1N1 Pandemic , 2011, PloS one.

[29]  Stefan Lee,et al.  Predicting Geo-informative Attributes in Large-Scale Image Collections Using Convolutional Neural Networks , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[30]  Peter Kontschieder,et al.  The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Stefano Ermon,et al.  Predicting Economic Development using Geolocated Wikipedia Articles , 2019, KDD.

[32]  Stefano Ermon,et al.  Efficient Object Detection in Large Images Using Deep Reinforcement Learning , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[33]  Marta C. González,et al.  Using Convolutional Networks and Satellite Imagery to Identify Patterns in Urban Environments at a Large Scale , 2017, KDD.

[34]  Nathan D. Cahill,et al.  Robust Spatial Filtering With Graph Convolutional Neural Networks , 2017, IEEE Journal of Selected Topics in Signal Processing.