Temporal Incoherence-Free Video Retargeting Using Foreground Aware Extrapolation

Video retargeting is a method of adjusting the aspect ratio of a given video to the target aspect ratio. However, temporal incoherence of video contents, which can occur frequently by video retargeting, is the most dominant factor that degrades the quality of retargeted videos. Current methods to maintain temporal coherence use the entire frames of the input videos; however, these methods cannot be implemented as on-time systems because of their tremendous computational complexity. As far as we know, there is no existing on-time video retargeting method that can avoid spatial distortion while perfectly maintaining temporal coherence. In this paper, we propose a novel on-time video retargeting method that can perfectly maintain temporal coherence and prevent the spatial distortion by using only two consecutive input frames. In our method, the maximum a posteriori-based foreground aware-block matching is used for the extrapolation that extends the side area of a given video to adjust its aspect ratio to the target. To maintain the temporal coherence of the extended area, the result of block matching for backward warping-based extrapolation of the start frame after the scene change occurs, is reused for the other frames until the next scene change occurs. In addition, we propose a scene scenario-adaptive fallback scheme to prevent severe distortions that can occur with reusing block matching results or extrapolation-based side extension. The simulation results showed that the proposed method greatly improved the bidirectional similarity value, which can measure the quality of video retargeting, by up to 10.26 compared with the existing on-time video retargeting methods.