Consumer video structuring by probabilistic merging of video segments

Accessing, organizing, and manipulating home videos constitutes a technical challenge due to their unrestricted content and the lack of storyline. In this paper, we present a methodology for structuring consumer video, based on the development of statistical models of similarity and adjacency between video segments in a probabilistic formulation. Learned Gaussian mixture models of inter-segment visual similarity, temporal adjacency, and segment duration are used to represent the classconditional densities of observed features. Such models are then used in a sequential merging algorithm consisting of a binary Bayes classifier, where the merging order is determined by a variation of Highest Confidence First (HCF), and the merging criterion is Maximum a Posteriori (MAP). The merging algorithm can be efficiently implemented and does not need any empirical parameter determination. Finally, the representation of the merging sequence by a tree provides for hierarchical, nonlinear access to the video content. Results on an eight-hour home video database illustrate the validity of our approach.