Video retargeting with multi-scale trajectory optimization

Mobile devices are increasingly powerful in media storage and rendering. The prevalent request of decent video browsing on mobile devices is demanding. However, one limitation comes from the size and aspect constraints of display. To display a video on a small screen, rendering process probably undergoes a sort of retargeting to fit into the target display and keep the most of original video information. In this paper, we formulate video retargeting as the problem of finding an optimal trajectory for a cropping window to go through the video, capturing the most salient region to scale towards proper display on the target. To measure the visual importance of every pixel, we utilize the local spatial-temporal saliency (ST-saliency) and face detection results. The spatiotemporal movement of the cropping window is modeled in a graph where a smoothed trajectory is resolved by a Max-Flow/Min-Cut method in a global optimization manner. Based on the horizontal/vertical projections and a graph-based method, the trajectory estimation of each shot can be conducted within one second. Also, the process of merging trajectories is employed to capture more saliency in video. Experimental results on diverse video contents have shown that our approach is efficient and subjective evaluation shows that the retargeted video has gained desirable user satisfaction.