Constrained fixation point based segmentation via deep neural network

Abstract It is an explicit mode to use the clicking points by the mouse in the interactive image segmentation, while an implicit interaction mode is to use the fixation points from the eye-tracking device. Both modes can provide a series of points. Inspired by the similarity between these two interaction modes, we propose a novel human visual system (HVS) based neural network for transferring the constrained fixation point based segmentation to the clicking point based interactive segmentation. Briefly speaking, the sequence of information transmission and processing in our model is RGB image, VGG-16 backbone, LGN-like module (LGNL) and ConvLSTM block, which correspond to the pathway of stimulus transmission and processing, i.e. stimulus, retina, lateral geniculate nucleus (LGN) and visual cortex in the HVS. First, the RGB image is fed to the VGG-16 backbone to obtain the multiple-layer feature maps. Then the LGNL is adopted to effectively incorporate edge-aware features and semantic features from different layers of the VGG-16 backbone in multiple resolutions, so as to produce rich contextual features. Finally, with the guidance of the fixation density map transformed from the fixation points, the output feature maps of LGNL are utilized to generate the segmentation map via a stack of ConvLSTM blocks in a coarse-to-fine manner. Comprehensive experiments demonstrate that the proposed HVS based neural network achieves a higher segmentation performance and outperforms seven state-of-the-art methods, and prove that the transfer from constrained fixation points to clicking points is reasonable and valid.