Improving Spatiotemporal Inpainting with Layer Appearance Models

The problem of removing blemishes in mosaics of building facades caused by foreground objects such as trees may be framed in terms of inpainting. Affected regions are first automatically segmented and then inpainted away using a combination of cues from unoccluded, temporally adjacent views of the same building patch, as well as surrounding unoccluded patches in the same frame. Discriminating the building layer from those containing foreground features is most directly accomplished through parallax due to camera motion over the sequence. However, the intricacy of tree silhouettes often complicates accurate motion-based segmentation, especially along their narrower branches. In this work we describe methods for automatically training appearance-based classifiers from a coarse motion-based segmentation to recognize foreground patches in static imagery and thereby improve the quality of the final mosaic. A local technique for photometric adjustment of inpainted patches which compensates for exposure variations between frames is also discussed.

[1]  Antonio Criminisi,et al.  Object categorization by learned universal visual dictionary , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[2]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[4]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[5]  Andrew Zisserman,et al.  A Statistical Approach to Texture Classification from Single Images , 2004, International Journal of Computer Vision.

[6]  Emanuele Trucco,et al.  Title : Making Good Features to Track Better , 1998 .

[7]  Jitendra Malik,et al.  Recognizing surfaces using three-dimensional textons , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Tai-Pang Wu,et al.  Video repairing: inference of foreground and background under severe occlusion , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[9]  Christopher Rasmussen,et al.  Spatiotemporal inpainting for recovering texture maps of partially occluded building facades , 2005, IEEE International Conference on Image Processing 2005.

[10]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[11]  Andrew Zisserman,et al.  Computer vision applied to super resolution , 2003, IEEE Signal Process. Mag..

[12]  Louis Laborelli,et al.  Missing data correction in still images and image sequences , 2002, MULTIMEDIA '02.

[13]  Gregory D. Hager,et al.  A two level approach for scene recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[15]  Marc Pollefeys,et al.  Radiometric alignment of image sequences , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[16]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[18]  Christopher Rasmussen,et al.  PCA-Based Recognition for Efficient Inpainting , 2006, ACCV.

[19]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[20]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[21]  Stefano Soatto,et al.  Real-time feature tracking and outlier rejection with changes in illumination , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.