Cascaded ConvLSTMs Using Semantically-Coherent Data Synthesis for Video Object Segmentation

This paper proposes a simple yet effective and efficient method for video object segmentation. Most existing methods take the color image and the optical flow as input for discovering the salient object in terms of appearance and motion. We instead leverage a ResNet backbone as an appearance-characterization encoder for each frame at different scales, and a series of Convolutional Long Short-Term Memory units (ConvLSTMs) as a motion-modeling decoder at each corresponding scale. By imposing supervision over each scale, such modules can well tackle all scales of a moving object with an inevitable scale variance over time. Instead of following a Condition Random Fields based post-processing, we use a more effective and efficient cascade module to refine the model predictions. Most existing video object segmentation datasets have limited sizes because it is expensive and time-consuming to obtain pixel-wise annotations. To overcome the data-insufficiency issue when training the deep network, we propose a semantically-coherent data synthesis strategy to augment training sequences without any efforts. Extensive experiments and ablation studies on the DAVIS 2016 dataset validate our proposed method. Furthermore, our method without the cascade module achieves a real-time speed of 26 fps on a single GPU.