Image completion with structure propagation

In this paper, we introduce a novel approach to image completion, which we call structure propagation. In our system, the user manually specifies important missing structure information by extending a few curves or line segments from the known to the unknown regions. Our approach synthesizes image patches along these user-specified curves in the unknown region using patches selected around the curves in the known region. Structure propagation is formulated as a global optimization problem by enforcing structure and consistency constraints. If only a single curve is specified, structure propagation is solved using Dynamic Programming. When multiple intersecting curves are specified, we adopt the Belief Propagation algorithm to find the optimal patches. After completing structure propagation, we fill in the remaining unknown regions using patch-based texture synthesis. We show that our approach works well on a number of examples that are challenging to state-of-the-art techniques.

[1]  K. Koffka.  Principles Of Gestalt Psychology , 1936 .

[2]  R. Bellman.  Dynamic programming. , 1957, Science.

[3]  Judea Pearl,et al.  Chapter 2 – BAYESIAN INFERENCE , 1988 .

[4]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[5]  Lucas Pereira,et al.  Image replacement through texture synthesis , 1997, Proceedings of International Conference on Image Processing.

[6]  L. Pessoa,et al.  Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception. , 1998, The Behavioral and brain sciences.

[7]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[9]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[10]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[11]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[12]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[13]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[14]  Tony F. Chan,et al.  Nontexture Inpainting by Curvature-Driven Diffusions , 2001, J. Vis. Commun. Image Represent..

[15]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[16]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[17]  Paul Harrison,et al.  A Non-Hierarchical Procedure for Re-Synthesis of Complex Textures , 2001, WSCG.

[18]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[19]  Louis Laborelli,et al.  Missing data correction in still images and image sequences , 2002, MULTIMEDIA '02.

[20]  William A. Barrett,et al.  Object-based image editing , 2002, ACM Trans. Graph..

[21]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[23]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[24]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[25]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[26]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[27]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[28]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[29]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[30]  D. Cohen-Or,et al.  Context-based surface completion , 2004, SIGGRAPH 2004.

[31]  Andrew Blake,et al.  PatchWorks: Example-Based Region Tiling for Image Editing , 2004 .

[32]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[33]  Silong Peng,et al.  Digital image inpainting using Monte Carlo method , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[34]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.