Semantic filtering for video stabilization

Moving objects pose a challenge to every video stabilization algorithm. We present a novel, efficient filtering technique that manages to remove outlier motion vectors caused from moving objects in a per-pixel smoothing setting. We leverage semantic information to change the calculation of optical flow, forcing the outliers to reside in the edges of our semantic mask. After a ‘content-preserving warping’ and a smoothing step we manage to produce stable and artifact-free videos.

[1]  Raanan Fattal,et al.  Video stabilization using epipolar geometry , 2012, TOGS.

[2]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Minh N. Do,et al.  Semantic Image Inpainting with Perceptual and Contextual Losses , 2016, ArXiv.

[4]  Michael Gleicher,et al.  Re-cinematography: improving the camera dynamics of casual video , 2007, ACM Multimedia.

[5]  Jian Sun,et al.  MeshFlow: Minimum Latency Online Video Stabilization , 2016, ECCV.

[6]  Michael Gleicher,et al.  Subspace video stabilization , 2011, TOGS.

[7]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Yuzhen Niu,et al.  Joint Subspace Stabilization for Stereoscopic Video , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  John Flynn,et al.  Deep Stereo: Learning to Predict New Views from the World's Imagery , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[11]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[13]  K. A. Semendyayev,et al.  Handbook of mathematics , 1985 .

[14]  Jian Sun,et al.  SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Bing-Yu Chen,et al.  Capturing Intention‐based Full‐Frame Video Stabilization , 2008, Comput. Graph. Forum.

[16]  Zihan Zhou,et al.  Plane-Based Content Preserving Warps for Video Stabilization , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Jiajun Bu,et al.  Video stabilization with a depth camera , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Jian Sun,et al.  Bundled camera paths for video stabilization , 2013, ACM Trans. Graph..

[19]  Feng Liu,et al.  Spatially and Temporally Optimized Video Stabilization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[20]  Philip H. S. Torr,et al.  Higher Order Conditional Random Fields in Deep Neural Networks , 2015, ECCV.

[21]  Michael J. Black,et al.  Optical Flow with Semantic Segmentation and Localized Layers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Eugenio Culurciello,et al.  ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation , 2016, ArXiv.

[23]  Irfan A. Essa,et al.  Auto-directed video stabilization with robust L1 optimal camera paths , 2011, CVPR 2011.

[24]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).