SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization

We propose a novel motion model, SteadyFlow, to represent the motion between neighboring video frames for stabilization. A SteadyFlow is a specific optical flow by enforcing strong spatial coherence, such that smoothing feature trajectories can be replaced by smoothing pixel profiles, which are motion vectors collected at the same pixel location in the SteadyFlow over time. In this way, we can avoid brittle feature tracking in a video stabilization system. Besides, SteadyFlow is a more general 2D motion model which can deal with spatially-variant motion. We initialize the SteadyFlow by optical flow and then discard discontinuous motions by a spatial-temporal analysis and fill in missing regions by motion completion. Our experiments demonstrate the effectiveness of our stabilization on real-world challenging videos.

[1]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[2]  Irfan A. Essa,et al.  Calibration-free rolling shutter removal , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[3]  Michael Gleicher,et al.  Subspace video stabilization , 2011, TOGS.

[4]  Jian Sun,et al.  Bundled camera paths for video stabilization , 2013, ACM Trans. Graph..

[5]  David Jacobs,et al.  CTSR 2011-03 Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes , 2011 .

[6]  Bing-Yu Chen,et al.  Capturing Intention‐based Full‐Frame Video Stabilization , 2008, Comput. Graph. Forum.

[7]  Jiajun Bu,et al.  Video stabilization with a depth camera , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Bing-Yu Chen,et al.  Video stabilization using robust feature trajectories , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[10]  Allen R. Hanson,et al.  Coherent Motion Segmentation in Moving Camera Videos Using Optical Flow Orientations , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Patrick Pérez,et al.  Multi-step flow fusion: towards accurate and dense correspondences in long video shots , 2012, BMVC.

[12]  Zihan Zhou,et al.  Plane-Based Content Preserving Warps for Video Stabilization , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Raanan Fattal,et al.  Video stabilization using epipolar geometry , 2012, TOGS.

[14]  Yuzhen Niu,et al.  Joint Subspace Stabilization for Stereoscopic Video , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  Rama Chellappa,et al.  Evaluation of image stabilization algorithms , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[16]  Michael Gleicher,et al.  Re-cinematography: improving the camera dynamics of casual video , 2007, ACM Multimedia.

[17]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[18]  Hailin Jin,et al.  Light field video stabilization , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[19]  Jitendra Malik,et al.  Motion segmentation and tracking using normalized cuts , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[20]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[21]  Irfan A. Essa,et al.  Auto-directed video stabilization with robust L1 optimal camera paths , 2011, CVPR 2011.

[22]  Feng Liu,et al.  Spatially and Temporally Optimized Video Stabilization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[23]  Bodo Rosenhahn,et al.  Multi-scale Clustering of Frame-to-Frame Correspondences for Motion Segmentation , 2012, ECCV.

[24]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[25]  Richard Szeliski,et al.  Removing rolling shutter wobble , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.