Bundled camera paths for video stabilization

We present a novel video stabilization method which models camera motion with a bundle of (multiple) camera paths. The proposed model is based on a mesh-based, spatially-variant motion representation and an adaptive, space-time path optimization. Our motion representation allows us to fundamentally handle parallax and rolling shutter effects while it does not require long feature trajectories or sparse 3D reconstruction. We introduce the 'as-similar-as-possible' idea to make motion estimation more robust. Our space-time path smoothing adaptively adjusts smoothness strength by considering discontinuities, cropping size and geometrical distortion in a unified optimization framework. The evaluation on a large variety of consumer videos demonstrates the merits of our method.

[1]  Seungyong Lee,et al.  Video deblurring for hand-held cameras using patch-based synthesis , 2012, ACM Trans. Graph..

[2]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Bing-Yu Chen,et al.  Video stabilization using robust feature trajectories , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[4]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[5]  Alfred M. Bruckstein,et al.  Over-Parameterized Variational Optical Flow , 2007, International Journal of Computer Vision.

[6]  K. A. Semendyayev,et al.  Handbook of mathematics , 1985 .

[7]  Michael S. Brown,et al.  Constructing image panoramas using dual-homography warping , 2011, CVPR 2011.

[8]  Richard Szeliski,et al.  Systems and Experiment Paper: Construction of Panoramic Image Mosaics with Global and Local Alignment , 2000, International Journal of Computer Vision.

[9]  Michael Gleicher,et al.  Subspace video stabilization , 2011, TOGS.

[10]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[11]  David Jacobs,et al.  CTSR 2011-03 Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes , 2011 .

[12]  Junichi Nakamura,et al.  Image Sensors and Signal Processing for Digital Still Cameras , 2005 .

[13]  Hailin Jin,et al.  Light field video stabilization , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[14]  Raanan Fattal,et al.  Video stabilization using epipolar geometry , 2012, TOGS.

[15]  Rama Chellappa,et al.  Evaluation of image stabilization algorithms , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[16]  Michael Gleicher,et al.  Re-cinematography: improving the camera dynamics of casual video , 2007, ACM Multimedia.

[17]  Yasuyuki Matsushita,et al.  Smoothly varying affine stitching , 2011, CVPR 2011.

[18]  Takeo Igarashi,et al.  As-rigid-as-possible shape manipulation , 2005, ACM Trans. Graph..

[19]  Per-Erik Forssén,et al.  Rectifying rolling shutter video from hand-held devices , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Homer H. Chen,et al.  Analysis and Compensation of Rolling Shutter Effect , 2008, IEEE Transactions on Image Processing.

[21]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[22]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[23]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[24]  Richard Szeliski,et al.  Motion Estimation with Quadtree Splines , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Irfan A. Essa,et al.  Calibration-free rolling shutter removal , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[26]  Richard Szeliski,et al.  Construction of Panoramic Image Mosaics with Global and Local Alignment , 2001 .

[27]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[28]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[29]  Irfan A. Essa,et al.  Auto-directed video stabilization with robust L1 optimal camera paths , 2011, CVPR 2011.

[30]  Richard Szeliski,et al.  Removing rolling shutter wobble , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[31]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[32]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[33]  Bing-Yu Chen,et al.  Capturing Intention‐based Full‐Frame Video Stabilization , 2008, Comput. Graph. Forum.

[34]  Jiajun Bu,et al.  Video stabilization with a depth camera , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.