Video object inpainting: a scale-robust method

Abstract Video inpainting is the process of reconstructing damaged regions of corrupted frames. In this research, we raise a few issues in existing video inpainting systems. They are usually not robust to the change in the object scale and cannot handle large missing regions behind the moving object. In this attempt, we will address the above issues as following: first, we extract moving objects from the background and construct two mosaic images for each object, a small mosaic and a large mosaic image. The small mosaic is used to detect the amount of scale changes in the moving objects and the large one is used to inpaint partially or completely corrupted objects. We next place the inpainted moving foreground in its location and rescale the objects to their original scale. Finally, we combine the inpainted moving foreground and the background to obtain the corrected video. To speed up the process, we have utilised a multi-resolution approach so that the patch are initially matched in a coarse resolution and later are refined in a fine resolution. The results confirm the robustness of our method in handling the scale change of moving objects and large missing regions.

[1]  Mohsen Soryani,et al.  Exemplar-based video inpainting with large patches , 2010, Journal of Zhejiang University SCIENCE C.

[2]  Ramakant Nevatia,et al.  Self-calibration of a camera from video of a walking human , 2002, Object recognition supported by user interaction for service robots.

[3]  Dong Liu,et al.  Image Compression With Edge-Based Inpainting , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Xiaochun Cao,et al.  Video Completion for Perspective Camera Under Constrained Motion , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[5]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[6]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[7]  Mubarak Shah,et al.  Motion Layer Based Object Removal in Videos , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[8]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Jenq-Neng Hwang,et al.  Exemplar-Based Video Inpainting Without Ghost Shadow Artifacts by Maintaining Temporal Continuity , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[11]  Marcelo Bertalmío,et al.  FLUID DYNAMICS, AND IMAGE AND VIDEO INPAINTING , 2001 .

[12]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[13]  Jian-Feng Cai,et al.  A framelet-based image inpainting algorithm , 2008 .

[14]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[16]  Tao Ding,et al.  A Rank Minimization Approach to Video Inpainting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[17]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, 2006 International Conference on Image Processing.

[18]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[20]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[21]  Yasuyuki Matsushita,et al.  Video Completion by Motion Field Transfer , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[22]  M. Fathy,et al.  Real-time Background Modeling/Subtraction using Two-Layer Codebook Model , 2008 .

[23]  Yong-Sheng Chen,et al.  Video object inpainting using manifold-based action prediction , 2010, 2010 IEEE International Conference on Image Processing.

[24]  Roland Göcke,et al.  Automatic Parametrisation for an Image Completion Method Based on Markov Random Fields , 2007, 2007 IEEE International Conference on Image Processing.

[25]  Jian Zhao,et al.  Efficient Object-Based Video Inpainting , 2006, ICIP.

[26]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Eli Shechtman,et al.  Space-time video completion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[28]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[29]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[30]  Baoxin Li,et al.  Video Inpainting for Largely Occluded Moving Human , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[31]  Igor Chueshov,et al.  Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting , 2001 .

[32]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.