Social sensing from street-level imagery: A case study in learning spatio-temporal urban mobility patterns

Abstract Street-level imagery has covered the comprehensive landscape of urban areas. Compared to satellite imagery, this new source of image data has the advantage in fine-grained observations of not only physical environment but also social sensing. Prior studies using street-level imagery focus primarily on urban physical environment auditing. In this study, we demonstrate the potential usage of street-level imagery in uncovering spatio-temporal urban mobility patterns. Our method assumes that the streetscape depicted in street-level imagery reflects urban functions and that urban streets of similar functions exhibit similar temporal mobility patterns. We present how a deep convolutional neural network (DCNN) can be trained to identify high-level scene features from street view images that can explain up to 66.5% of the hourly variation of taxi trips along with the urban road network. The study shows that street-level imagery, as the counterpart of remote sensing imagery, provides an opportunity to infer fine-scale human activity information of an urban region and bridge gaps between the physical space and human space. This approach can therefore facilitate urban environment observation and smart urban planning.

[1]  M. Betsill,et al.  Rethinking Sustainable Cities: Multilevel Governance and the 'Urban' Politics of Climate Change , 2005 .

[2]  Bolei Zhou,et al.  Scene Parsing through ADE20K Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[4]  Jinhua Zhao,et al.  C-IMAGE: city cognitive mapping through geo-tagged photos , 2016 .

[5]  Xing Xie,et al.  Discovering regions of different functions in a city using human mobility and POIs , 2012, KDD.

[6]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Mani Golparvar-Fard,et al.  Segmentation and recognition of roadway assets from car-mounted camera video streams using a scalable non-parametric image parsing method , 2015 .

[8]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[9]  R. Ahas,et al.  Daily rhythms of suburban commuters' movements in the Tallinn metropolitan area: Case study with mobile positioning data , 2010 .

[10]  Hang-Bong Kang,et al.  Prediction of crime occurrence from multi-modal data using deep learning , 2017, PloS one.

[11]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Di Zhu,et al.  Street as a big geo-data assembly and analysis unit in urban studies: A case study using Beijing taxi data , 2017 .

[13]  Christian Früh,et al.  Google Street View: Capturing the World at Street Level , 2010, Computer.

[14]  Chenghu Zhou,et al.  A new insight into land use classification based on aggregated mobile phone data , 2013, Int. J. Geogr. Inf. Sci..

[15]  Yi Zhu,et al.  What is it like down there?: generating dense ground-level views and image features from overhead imagery using conditional generative adversarial networks , 2018, SIGSPATIAL/GIS.

[16]  Yu Liu,et al.  Extracting human emotions at different places based on facial expressions and spatial clustering analysis , 2019, Trans. GIS.

[17]  Liang Liu,et al.  Understanding individual and collective mobility patterns from smart card records: A case study in Shenzhen , 2009, 2009 12th International IEEE Conference on Intelligent Transportation Systems.

[18]  Ramesh Raskar,et al.  Computer vision uncovers predictors of physical urban change , 2017, Proceedings of the National Academy of Sciences.

[19]  Zbigniew Smoreda,et al.  Everyday space–time geographies: using mobile phone-based sensor data to monitor urban activity in Harbin, Paris, and Tallinn , 2015, Int. J. Geogr. Inf. Sci..

[20]  Daqing Zhang,et al.  Measuring social functions of city regions from large-scale taxi behaviors , 2011, 2011 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[21]  Xiaoping Liu,et al.  Sensing spatial distribution of urban land use by integrating points-of-interest and Google Word2Vec model , 2017, Int. J. Geogr. Inf. Sci..

[22]  Yuan Tian,et al.  Understanding intra-urban trip patterns from taxi trajectory data , 2012, Journal of Geographical Systems.

[23]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[24]  Michael Batty,et al.  Detecting the dynamics of urban structure through spatial network analysis , 2014, Int. J. Geogr. Inf. Sci..

[25]  Hanne Kirstine Adriansen,et al.  The geography of pastoral mobility: A spatio-temporal analysis of GPS data from Sahelian Senegal , 2005 .

[26]  Hui Lin,et al.  Representing place locales using scene elements , 2018, Comput. Environ. Urban Syst..

[27]  Bolei Zhou,et al.  Landscape and Urban Planning , 2018 .

[28]  Marc Barthelemy,et al.  A stochastic model of randomly accelerated walkers for human mobility , 2015, Nature Communications.

[29]  Sang Michael Xie,et al.  Combining satellite imagery and machine learning to predict poverty , 2016, Science.

[30]  Cecilia Mascolo,et al.  A Tale of Many Cities: Universal Patterns in Human Urban Mobility , 2011, PloS one.

[31]  Warren C. Jochem,et al.  Spatially disaggregated population estimates in the absence of national population and housing census data , 2018, Proceedings of the National Academy of Sciences.

[32]  John Scott Bridle,et al.  Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition , 1989, NATO Neurocomputing.

[33]  Marta C. González,et al.  Using Convolutional Networks and Satellite Imagery to Identify Patterns in Urban Environments at a Large Scale , 2017, KDD.

[34]  Ramesh Raskar,et al.  Streetscore -- Predicting the Perceived Safety of One Million Streetscapes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[35]  Y. Tuan Space and Place: Humanistic Perspective , 1979 .

[36]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Jonathan Krause,et al.  Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States , 2017, Proceedings of the National Academy of Sciences.

[38]  M. Batty,et al.  Gravity versus radiation models: on the importance of scale and heterogeneity in commuting flows. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[39]  Fahui Wang,et al.  Urban land uses and traffic 'source-sink areas': Evidence from GPS-enabled taxi data in Shanghai , 2012 .

[40]  Pablo Martí,et al.  Social Media data: Challenges, opportunities and limitations in urban studies , 2019, Comput. Environ. Urban Syst..

[41]  Yuhao Kang,et al.  MAPPING THE SENSITIVITY OF THE PUBLIC EMOTION TO THE MOVEMENT OF STOCK MARKET VALUE: A CASE STUDY OF MANHATTAN , 2017 .

[42]  Fahad Shahbaz Khan,et al.  Discriminative Color Descriptors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Bernt Schiele,et al.  Learning Deep Representations of Fine-Grained Visual Descriptions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Carlo Ratti,et al.  Discovering place-informative scenes and objects using social media photos , 2019, Royal Society Open Science.

[45]  Sergey Levine,et al.  Learning modular neural network policies for multi-task and multi-robot transfer , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[46]  J. Chanussot,et al.  Hyperspectral Remote Sensing Data Analysis and Future Challenges , 2013, IEEE Geoscience and Remote Sensing Magazine.

[47]  R. Stedman Is It Really Just a Social Construction?: The Contribution of the Physical Environment to Sense of Place , 2003 .

[48]  Yong Gao,et al.  Uncovering Patterns of Inter-Urban Trip and Spatial Interaction from Social Media Check-In Data , 2013, PloS one.

[49]  Edward Ng,et al.  Mapping sky, tree, and building view factors of street canyons in a high-density urban environment , 2018 .

[50]  T. M. Lillesand,et al.  Remote Sensing and Image Interpretation , 1980 .

[51]  Albert-László Barabási,et al.  Understanding individual human mobility patterns , 2008, Nature.

[52]  Jianxiong Xiao,et al.  What Makes a Photograph Memorable? , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Serge J. Belongie,et al.  Learning deep representations for ground-to-aerial geolocalization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Chaogui Kang,et al.  Social Sensing: A New Approach to Understanding Our Socioeconomic Environments , 2015 .

[56]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[57]  Cynthia Chen,et al.  Diurnal Pattern of Transit Ridership: Case Study of New York City Subway System , 2009 .

[58]  Byoungkwon An,et al.  Looking Beyond the Visible Scene , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Zhiqiang Shen,et al.  DSOD: Learning Deeply Supervised Object Detectors from Scratch , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).