Structure-preserving image completion with multi-level dynamic patches

In this paper, we present a novel structure-preserving image completion approach equipped with dynamic patches. We formulate the image completion problem into an energy minimization framework that accounts for coherence within the hole and global coherence simultaneously. The completion of the hole is achieved through iterative optimizations combined with a multi-scale solution. In order to avoid abnormal structure and disordered texture, we utilize a dynamic patch system to achieve efficient structure restoration. Our dynamic patch system functions in both horizontal and vertical directions of the image pyramid. In the horizontal direction, we conduct a parallel search for multi-size patches in each pyramid level and design a competitive mechanism to select the most suitable patch. In the vertical direction, we use large patches in higher pyramid level to maximize the structure restoration and use small patches in lower pyramid level to reduce computational workload. We test our approach on massive images with complex structure and texture. The results are visually pleasing and preserve nice structure. Apart from effective structure preservation, our approach outperforms previous state-of-the-art methods in time consumption.

[1]  Zhixiong Chen,et al.  On the k-error linear complexity of binary sequences derived from polynomial quotients , 2012, Science China Information Sciences.

[2]  Xuelong Li,et al.  Robust Match Fusion Using Optimization , 2015, IEEE Transactions on Cybernetics.

[3]  Wencheng Wang,et al.  Effective structure restoration for image completion using internet resources , 2015, The Visual Computer.

[4]  Zhijing Yang,et al.  Blind inpainting using the fully convolutional neural network , 2017, The Visual Computer.

[5]  Tony F. Chan,et al.  Euler's Elastica and Curvature-Based Inpainting , 2003, SIAM J. Appl. Math..

[6]  Mohammad H. Mahoor,et al.  Fast image blending using watersheds and graph cuts , 2009, Image Vis. Comput..

[7]  Enhua Wu,et al.  Image completion with dynamic patches , 2017, CGI.

[8]  Xuelong Li,et al.  Exposure Fusion Using Boosting Laplacian Pyramid , 2014, IEEE Transactions on Cybernetics.

[9]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[10]  Brian Bouzas,et al.  Objective image quality measure derived from digital image power spectra , 1992 .

[11]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[12]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[13]  Enhua Wu,et al.  Image completion with perspective constraint based on a single image , 2015, Science China Information Sciences.

[14]  Hiroshi Ishikawa,et al.  Globally and locally consistent image completion , 2017, ACM Trans. Graph..

[15]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..

[16]  D. R. Fulkerson,et al.  Flows in Networks. , 1964 .

[17]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[18]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[19]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[20]  Micah K. Johnson,et al.  Multi-scale image harmonization , 2010, ACM Trans. Graph..

[21]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[22]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Wojciech Matusik,et al.  Multi-scale image harmonization , 2010, SIGGRAPH 2010.

[24]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[25]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[26]  Ross T. Whitaker A level-set approach to image blending , 2000, IEEE Trans. Image Process..

[27]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[28]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[29]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[30]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[31]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[32]  Guillermo Sapiro,et al.  A Comprehensive Framework for Image Inpainting , 2010, IEEE Transactions on Image Processing.

[33]  Cláudio Rosito Jung,et al.  Block-based image inpainting in the wavelet domain , 2007, The Visual Computer.

[34]  Xuelong Li,et al.  Structured-Patch Optimization for Dense Correspondence , 2015, IEEE Transactions on Multimedia.