Full-frame video stabilization with motion inpainting

Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality. While most previous methods end up with producing smaller size stabilized videos, our completion method can produce full-frame videos by naturally filling in missing image parts by locally aligning image data of neighboring frames. To achieve this, motion inpainting is proposed to enforce spatial and temporal consistency of the completion in both static and dynamic image areas. In addition, image quality in the stabilized video is enhanced with a new practical deblurring algorithm. Instead of estimating point spread functions, our method transfers and interpolates sharper image pixels of neighboring frames to increase the sharpness of the frame. The proposed video completion and deblurring methods enabled us to develop a complete video stabilizer which can naturally keep the original image quality in the stabilized videos. The effectiveness of our method is confirmed by extensive experiments over a wide variety of videos

[1]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[2]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[3]  M. Carter.  Computer graphics: Principles and practice , 1997 .

[4]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[5]  P. Jansson.  Deconvolution of images and spectra (2nd ed.) , 1996 .

[6]  Shree K. Nayar,et al.  Video super-resolution using controlled subpixel detector shifts , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Brendan J. Frey,et al.  Video Epitomes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[8]  Electronic Imaging: Image and Video Communications and Processing 2005, San Jose, California, USA, 16-20 January 2005 , 2005, Electronic Imaging: Image and Video Communications and Processing.

[9]  Richard Szeliski,et al.  Construction of panoramic mosaics with global and lo-cal alignment , 2020 .

[10]  Eli Shechtman,et al.  Space-time video completion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[11]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[12]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[13]  Maurizio Pilu,et al.  Video stabilization as a variational problem and numerical solution with the Viterbi method , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[14]  Shmuel Peleg,et al.  Restoration of multiple images with motion blur in different directions , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[15]  Richard Szeliski,et al.  Construction of Panoramic Image Mosaics with Global and Local Alignment , 2001 .

[16]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[17]  Yitzhak Yitzhaky,et al.  Restoration of an image degraded by vibrations using only a single frame , 2000 .

[18]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[19]  J. Sethian.  Level set methods : evolving interfaces in geometry, fluid mechanics, computer vision, and materials science , 1996 .

[20]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[21]  Andrew Zisserman,et al.  Automated mosaicing with super-resolution zoom , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[22]  Andrew Blake,et al.  Motion Deblurring and Super-resolution from an Image Sequence , 1996, ECCV.

[23]  David Malah,et al.  Robust identification of motion and out-of-focus blur parameters from blurred and noisy images , 1991, CVGIP Graph. Model. Image Process..

[24]  Andrea Fusiello,et al.  Image stabilization by features tracking , 1999, Proceedings 10th International Conference on Image Analysis and Processing.

[25]  James A. Sethian,et al.  Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid , 2012 .

[26]  Tat-Jen Cham,et al.  A statistical framework for long-range feature matching in uncalibrated image mosaicing , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[27]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[28]  Robert Strzodka,et al.  Real-time motion estimation and visualization on graphics cards , 2004, IEEE Visualization 2004.

[29]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[30]  A. Vladár,et al.  Image sharpness measurement in the scanning electron-microscope--part III. , 2006, Scanning.

[31]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[32]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[33]  Janusz Konrad,et al.  Probabilistic video stabilization using Kalman filtering and mosaicing , 2003, IS&T/SPIE Electronic Imaging.

[34]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[35]  Eric Krotkov,et al.  Focusing , 2004, International Journal of Computer Vision.

[36]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[37]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[38]  Rachid Deriche,et al.  Using geometric corners to build a 2D mosaic from a set of images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  Shree K. Nayar,et al.  Motion-based motion deblurring , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Richard Szeliski,et al.  Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment , 2000, International Journal of Computer Vision.

[41]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[42]  Zhigang Zhu.  CAMERA STABILIZATION BASED ON 2.5D MOTION ESTIMATION AND INERTIAL MOTION FILTERING , 1998 .

[43]  Richard Szeliski,et al.  Image mosaicing for tele-reality applications , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[44]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[45]  Azriel Rosenfeld,et al.  Shooting a smooth video with a shaky camera , 2003, Machine Vision and Applications.

[46]  P. Anandan,et al.  A computational framework and an algorithm for the measurement of visual motion , 1987, International Journal of Computer Vision.