A Fusion Approach to Grayscale-Thermal Tracking with Cross-Modal Sparse Representation

Grayscale-thermal tracking receives much attention recently due to the complementary benefits of the visible and thermal infrared modalities in over- coming the imaging limitations of individual source. This paper investigates how to perform effective fusion of the grayscale and thermal information for robust object tracking. We propose a novel fusion approach based on the cross-modal sparse representation in the Bayesian filtering framework. First, to exploit the interdependence of different modalities, we take both the intra- and inter-modality constraints into account in the sparse representation, i.e., cross-modal sparse rep- resentation. Moreover, we introduce the modality weights in our model to achieve adaptive fusion. Second, unlike conventional methods, we employ the reconstruction residues and coefficients together to define the likelihood probability for each candidate sample generated by the motion model. Finally, the object is located by finding the candidate sample with the maximum likelihood probability. Experimental results on the public benchmark dataset suggest that the proposed approach performs favourably against the state-of-the-art grayscale-thermal trackers.