A fusion network for road detection via spatial propagation and spatial transformation

Abstract In this paper, we address the fusion of image and point cloud data for road detection. To take advantage of both deep network and multi-modal data fusion, we propose an end-to-end road segmentation network called SPSTFN (Spatial Propagation and Spatial Transformation Fusion Network). Our method considers the model-level fusion and dual-view fusion in the network simultaneously for the first time. Specifically, the proposed SPSTFN contains three parts: the point cloud branch, the image branch, and the fusion block. Firstly, we design a simple but efficient lightweight network to handle the unordered and sparse point cloud to obtain a coarse representation of the road area. Then, an equal-resolution convolutional block is adopted to capture the low-level features of the image which are used to produce the heat diffusion coefficients of the joint anisotropic diffusion based spatial propagation model. Thirdly, we conduct the diffusion process on the coarse representation under the guidance of the learned low-level image features, both in the perspective and bird views, via the spatial transformation in the network. Finally, the diffusion results of the two views are then integrated to generate the final refined representation of the road area. The proposed fusion method is totally data-driven and parameter-free, and the whole fusion network can be trained with the standard BP (Back Propagation) algorithm. Without any additional process steps and pre-training, the proposed method obtains competitive results on the KITTI Road Benchmark.