Sidewalk extraction using aerial and street view images

A reliable, punctual, and spatially accurate dataset of sidewalks is vital for identifying where improvements can be made upon urban environment to enhance multi-modal accessibility, social cohesion, and residents' physical activity. This paper develops a synthetically new spatial procedure to extract the sidewalk by integrating the detected results from aerial and street view imagery. We first train neural networks to extract sidewalks from aerial images, and then use pre-trained models to restore occluded and missing sidewalks from street view images. By combining the results from both data sources, a complete network of sidewalks can be produced. Our case study includes four counties in the U.S., and both precision and recall reach about 0.9. The street view imagery helps restore the occluded sidewalks and largely enhances the sidewalk network's connectivity by linking 20% of dangles.

[1]  IEEE conference on computer vision and pattern recognition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[2]  A. Loukaitou-Sideris.  Is It Safe to Walk Here? Design and Policy Responses to Women's Fear of Victimization in Public Places , 2005 .

[3]  Ann Forsyth,et al.  Cities Afoot—Pedestrians, Walkability and Urban Design , 2008 .

[4]  Mohamad Awad,et al.  An Unsupervised Artificial Neural Network Method for Satellite Image Segmentation , 2010, Int. Arab J. Inf. Technol..

[5]  Y. Zhang,et al.  A REVIEW ON IMAGE SEGMENTATION TECHNIQUES WITH REMOTE SENSING PERSPECTIVE , 2010 .

[6]  Dong-Chen He,et al.  Automatic change detection of buildings in urban environment from very high spatial resolution images using existing geodatabase and prior knowledge , 2010 .

[7]  Jaisung Choi,et al.  Determining the sidewalk pavement width by using pedestrian discomfort levels and movement characteristics , 2011 .

[8]  Ahmed M. Elgammal,et al.  Segmentation of occluded sidewalks in satellite images , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[9]  Ahmed M. Elgammal,et al.  Satellite image based precise robot localization on sidewalks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[10]  Jitendra Malik,et al.  Classification of sidewalks in street view images , 2013, 2013 International Green Computing Conference Proceedings.

[11]  Chen Zheng,et al.  Semantic Segmentation of Remote Sensing Imagery Using Object-Based Markov Random Field Model With Regional Penalties , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[12]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[13]  Marco Cavallo.  3D City Reconstruction From Google Street View , 2015 .

[14]  Luisa Verdoliva,et al.  Marker-Controlled Watershed-Based Segmentation of Multiresolution Remote Sensing Images , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Nikos Komodakis,et al.  Building detection in very high resolution multispectral data with deep learning features , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[16]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Sanja Fidler,et al.  HD Maps: Fine-Grained Road Segmentation by Parsing Ground and Aerial Images , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Yi Li,et al.  Fully Convolutional Instance-Aware Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Bolei Zhou,et al.  Scene Parsing through ADE20K Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Xiao Xiang Zhu,et al.  Deep learning in remote sensing: a review , 2017, ArXiv.

[23]  Tarek Sayed,et al.  Evaluating the impact of connectivity, continuity, and topography of sidewalk network on pedestrian safety. , 2017, Accident; analysis and prevention.

[24]  Stuart E. Marsh,et al.  Designing healthy communities: Testing the walkability model , 2017 .

[25]  Nataliia Kussul,et al.  Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data , 2017, IEEE Geoscience and Remote Sensing Letters.

[26]  Derek T. Anderson,et al.  Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community , 2017 .

[27]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[29]  Chao Wu,et al.  Using the TensorFlow Deep Neural Network to Classify Mainland China Visitor Behaviours in Hong Kong from Check-in Data , 2018, ISPRS Int. J. Geo Inf..

[30]  Ilya Afanasyev,et al.  Deep Learning Approach for Building Detection in Satellite Multispectral Imagery , 2018, 2018 International Conference on Intelligent Systems (IS).

[31]  Hui Kong,et al.  Curb Detection for Road and Sidewalk Detection , 2018, IEEE Transactions on Vehicular Technology.

[32]  Yinghua Ye,et al.  A Deep Learning Approach on Building Detection from Unmanned Aerial Vehicle-Based Images in Riverbank Monitoring , 2018, Sensors.

[33]  Xia Li,et al.  Road Detection From Remote Sensing Images by Generative Adversarial Networks , 2018, IEEE Access.

[34]  Wei Tu,et al.  Integrating Aerial and Street View Images for Urban Land Use Classification , 2018, Remote. Sens..

[35]  Bolei Zhou,et al.  Places: A 10 Million Image Database for Scene Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Buyu Liu,et al.  A Parametric Top-View Representation of Complex Road Scenes , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Xiao Xiang Zhu,et al.  Model Fusion for Building Type Classification from Aerial and Street View Images , 2019, Remote. Sens..

[38]  Ujjwal Maulik,et al.  Understanding Deep Learning Techniques for Image Segmentation , 2019, ACM Comput. Surv..

[39]  Yang Chen,et al.  Road Extraction from High-Resolution Remote Sensing Imagery Using Refined Deep Residual Convolutional Neural Network , 2019, Remote. Sens..

[40]  Pietro Perona,et al.  From Google Maps to a Fine-Grained Catalog of Street trees , 2019, ArXiv.

[41]  Guoyuan Wu,et al.  Developing an Aerial-Image-Based Approach for Creating Digital Sidewalk Inventories , 2019 .

[42]  Yong Jae Lee,et al.  YOLACT: Real-Time Instance Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  Miguel Figliozzi,et al.  Study of Sidewalk Autonomous Delivery Robots and Their Potential Impacts on Freight Efficiency and Travel , 2019, Transportation Research Record: Journal of the Transportation Research Board.

[44]  Jun Rao,et al.  Road Segmentation Based on Hybrid Convolutional Network for High-Resolution Visible Remote Sensing Image , 2019, IEEE Geoscience and Remote Sensing Letters.

[45]  Gang Li,et al.  Profiling unmanned aerial vehicle photography tourists , 2020, Current Issues in Tourism.

[46]  Cuizhen Wang,et al.  Detecting new building construction in urban areas based on images of small unmanned aerial system , 2020 .

[47]  Cuizhen Wang,et al.  Choosing an appropriate training set size when using existing data to train neural networks for land cover segmentation , 2020, Ann. GIS.

[48]  Libertad Rizo-Corona,et al.  The role of perceived pedestrian safety on designing safer built environments , 2020, Traffic injury prevention.

[49]  Chengbo Ai,et al.  A network-level sidewalk inventory method using mobile LiDAR and deep learning , 2020 .

[50]  Dong Li,et al.  Evaluating the effect of city lock-down on controlling COVID-19 propagation through deep learning and network science models , 2020, Cities.

[51]  Pourya Shamsolmoali,et al.  Road Segmentation for Remote Sensing Images Using Adversarial Spatial Pyramid Networks , 2020, IEEE Transactions on Geoscience and Remote Sensing.

[52]  A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images , 2019, Applied Artificial Intelligence.