Multi-view based neural network for semantic segmentation on 3D scenes

Dear editor, For semantic segmentation tasks, convolutional neural network (CNN) based methods have been prevalent for both 2D image semantic segmentation and 3D semantic segmentation. Though traditional methods often use local features to segment a target (For example, in [1] both 2D local features and 3D local features are used to boost recognition ability), CNN based methods [2,3] exhibit much better performance than traditional methods [4]. In all the CNN based methods on images, fully convolutional networks (FCNs) [2] are firstly proposed for end-to-end training. Basically, all the following methods are the variants of FCNs. For 3D input, some studies leverage 3D convolution to predict dense 3D semantic voxel maps [5]. However, 3D convolution has the limitation of low resolution as the GPU memory constraint. Additionally, RGB information is not well considered though it is very important. As semantic segmentation on images has been very good, we can project the semantic segmentation results of images to 3D mesh based on the geometric relationship. In this study, we mainly exploit the multi-view based neural network for semantic segmentation on 3D scenes.