Joint spatiograms for multi-modality tracking with online update

Abstract Integrating multiple different yet complementary modalities has been proved to be an effective way for boosting tracking performance. Many previous works just consider the fusion of different features from identical image or identical features from images with different spectrum alone, which makes them be quite distinct from each other and be hard to be integrated naturally. In this study, we propose a unified tracking framework to integrate multiple different modalities via innovative use of spatiogram, where the spatiogram is formed by weighting each bin of histogram with the mean and covariance of the locations of the pixels that contribute to that bin. Specifically, each modal target and its candidate are first represented by second-order spatiogram, and their similarity is measured by the weighted Bhattacharayya coefficient. Next, an objective function is built by integrating all modal similarities, then a joint center-shift formula of the target is gained by performing Taylor expansion and gradient minimization on the objective function. Finally, the optimal target location is gained recursively by applying the mean shift procedure. Besides, a fast fuzzy logic system is designed to adaptively adjust the weight of each modality, and a model update scheme based on particle filter is developed to capture the appearance variations. Our framework allows the modality to be original gray of pixel or other extracted feature from single image or different spectral images, and provides the flexibility to arbitrarily add or remove modality. Experimental results on three challenging public datasets demonstrate clearly the robustness and effectiveness of the proposed method.