A Framework to Combine Multi-Object Video Segmentation and Tracking

Multi-object video segmentation and multi-object tracking are very similar in the aspect that both determine the locations and maintain the identities of the objects of interest (targets) in each frame of the video. Our approach takes advantage of this fact and uses the strengths of one task to improve the accuracy of the other. In our framework, the multi-object tracking and segmentation modules initially produce results on our dataset independently. The tracking module enforces higher-order smoothness constraints on the object trajectories and uses Lagrangian relaxation to get an iterative solution method. The segmentation module forms superpixels through clustering, trains a linear SVM using Lab color to obtain the foreground and background segmentation and assigns ID labels based on color and optical flow. The results of these two modules are then jointly processed and updated. The locations of the tracking bounding boxes are refined with the help of the segmentation results, so that they are more precisely centered on the targets. The tracking module is more accurate in terms of ID assignment and hence, its results are used to correct errors in ID labeling in the segmentation module. Both modules identify and add any target detections they initially missed to their results using the results of the other component. Hence, this joint processing increases the accuracy of both the tracking and the segmentation results as can be seen from our experimental results. Our approach is comparable to state-of-the-art tracking and segmentation techniques.