Video segmentation with motion smoothness

In this extended abstract, we propose a novel approach for video segmentation by utilizing motion information. Recently, graph-cutbased segmentation methods became popular in this domain but most of them dealt with color information only. Those methods possibly fail if there are regions similar in color between foreground and background. Unfortunately, it is usually hard to avoid, especially when objects are filmed under a natural environment. For instance, Figure 1(a) shows a result of graph cut with a small smoothness weighting, and hence some background regions are incorrectly labeled. On the contrary, if a larger smoothness weighting is used, some background regions near the foreground will be merged as shown in Figure 1(b). To improve those drawbacks, we propose a method based on both of color and motion information to conduct the segmentation. The method is useful because foreground and background usually have different motion patterns as shown in Figure 1(c).

[1]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Takashi Totsuka,et al.  AutoKey: human assisted key extraction , 1995, SIGGRAPH.

[3]  Michael Isard,et al.  Active Contours , 2000, Springer London.

[4]  Mo Chen,et al.  Progressive cut , 2006, MM '06.

[5]  Michael Gleicher,et al.  This document was created with FrameMaker 4.0.4 Image Snapping , 2022 .

[6]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[7]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Harry Shum,et al.  Video object cut and paste , 2005, ACM Trans. Graph..

[9]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.

[10]  William A. Barrett,et al.  Intelligent scissors for image composition , 1995, SIGGRAPH.

[11]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[13]  Bing-Yu Chen,et al.  Capturing Intention‐based Full‐Frame Video Stabilization , 2008, Comput. Graph. Forum.

[14]  Mubarak Shah,et al.  Motion layer extraction in the presence of occlusion using graph cuts , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Guillermo Sapiro,et al.  A Geodesic Framework for Fast Interactive Image and Video Segmentation and Matting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[17]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, ACM Trans. Graph..

[18]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..