Intrackability: Characterizing Video Statistics and Pursuing Video Representations

Videos of natural environments contain a wide variety of motion patterns of varying complexities which are represented by many different models in the vision literature. In many situations, a tracking algorithm is formulated as maximizing a posterior probability. In this paper, we propose to measure the video complexity by the entropy of the posterior probability, called the intrackability, to characterize the video statistics and pursue optimal video representations. Based on the definition of intrackability, our study is aimed at three objectives. Firstly, we characterize video clips of natural scenes by intrackability. We calculate the intrackabilities of image points to measure the local inferential uncertainty, and collect the histogram of the intrackabilities over the video in space and time as the global video statistics. We find that a PCA scatter-plot based on the first two principle components of intrackability histograms can reflect the major variations, i.e., image scaling and object density, in natural video clips. Secondly, we show that different video representations, including deformable contours, tracking kernels with various appearance features, dense motion fields, and dynamic texture models, are connected by the change of intrackability and thus develop a simple criterion for model transition and for pursuing the optimal video representation. Thirdly, we derive the connections between the intrackability measure and other criteria in the literature such as the Shi-Tomasi texturedness measure, conditional number, and Harris-Stephens R score, and compare with the Shi-Tomasi measure in tracking experiments.