Detecting individual abandoned houses from google street view: A hierarchical deep learning approach

Abstract Abandoned houses (AH) are focal points in urban communities by threatening local security, destroying housing markets, and burdening government finance in the U.S. legacy cities. In particular, individual-level AH detection provides essential information for fine-resolution urban studies, government decision-makers, and private sector practitioners. However, three primary conventional data sources (field data, utility data, and remote sensing data) cannot suffice to collect such fine-resolution data in the large spatial area via a cost-effective approach. To this end, Google Street View (GSV) imagery, which emerges as the mainstream open-access data source with global coverage, provides an opportunity to address this issue. Subsequently, a follow-up challenge confronting the detection of AH arises from the fact that it lacks an effective method that can discern authentic visual features from the redundant noise in GSV images. In this study, we aim to develop an effective method to detect individual-level AH from GSV imagery. Specifically, we developed a new hierarchical deep learning method to leverage both global and local visual features of AH in the detection. The method can be further divided into three steps: (1) Scene-based classification that can extract global visual features of AH was implemented through fine-tuning a pre-trained deep convolutional neural network (CNN) model. (2) We developed a patch-based classification method that can extract specific local features of AH. In this method, patches were generated from GSV images based on auto-detected local features, followed by being labeled as three categories: building patches, vegetation patches, and others. Two deep CNN models were employed to identify deteriorated building facade patches and overgrown vegetation patches, respectively. (3) Individual-level AH were detected by integrating scene classification results and patch classification results in a decision-tree model. Experimental results showed that the F-score of AH was 0.84 in a well-prepared dataset collected from five different Rust Belt cities. The proposed hierarchical deep learning approach effectively improved the accuracy comparing with the traditional scene-based method. In addition, the proposed method was applied to generate an AH map in a new site in Detroit, MI. Our study demonstrated the feasibility of GSV imagery in AH detection and showed great potential to detect AH in a large spatial extent.