Vector ordering based multimodal video skimming for user videos

Video skimming is generation of a shorter video as a summary for any given video, containing a subset of its segments that are sufficient to convey its purpose. User videos, which are often almost structureless, do not have any predefined script or events to help in summarization. Use of multiple modalities with a proper fusion strategy would be beneficial for skimming of such videos. In this paper, first, r(educed)-ordering based importance ranking of video segments is performed on audio and visual channels independently. A round robin based fusion scheme is proposed for combining importance ranks generated considering multiple modalities, and applied on the importance ranks from audio and visual channels. The fused rank is then used to generate the video summary. Experimental results show that the proposed fusion scheme outperforms relevant low level fusion and single modality cases, when r-ordering-based and other schemes are used for importance determination in each modality.