Image Completion Approaches Using the Statistics of Similar Patches

Image completion involves filling missing parts in images. In this paper we address this problem through novel statistics of similar patches. We observe that if we match similar patches in the image and obtain their offsets (relative positions), the statistics of these offsets are sparsely distributed. We further observe that a few dominant offsets provide reliable information for completing the image. Such statistics can be incorporated into both matching-based and graph-based methods for image completion. Experiments show that our method yields better results in various challenging cases, and is faster than existing state-of-the-art methods.

[1]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[2]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[4]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[5]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[7]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[8]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[9]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[10]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[12]  E. Shechtman,et al.  Transactions on Pattern Analysis and Machine Intelligence 1 Space-time Video Completion Draft Transactions on Pattern Analysis and Machine Intelligence 2 , 2022 .

[13]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[14]  Chi-Keung Tang,et al.  Inference of segmented color and texture description by tensor voting , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[16]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[17]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[18]  Nikos Komodakis,et al.  Image Completion Using Global Optimization , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[19]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Eli Shechtman,et al.  Matching Local Self-Similarities across Images and Videos , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Nikos Komodakis,et al.  Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning , 2007, IEEE Transactions on Image Processing.

[22]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Michal Irani,et al.  What Is a Good Image Segment? A Unified Approach to Segment Extraction , 2008, ECCV.

[25]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[26]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[27]  Sundaresh Ram,et al.  Removing Camera Shake from a Single Photograph , 2009 .

[28]  Jaakko Astola,et al.  From Local Kernel to Nonlocal Multiple-Model Image Denoising , 2009, International Journal of Computer Vision.

[29]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[30]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[31]  Thomas Deselaers,et al.  Global and efficient self-similarity for object classification and detection , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Zeev Farbman,et al.  Convolution pyramids , 2011, ACM Trans. Graph..

[33]  Adam Finkelstein,et al.  Patchmatch: a fast randomized matching algorithm with application to image and video , 2011 .

[34]  Shai Avidan,et al.  Coherency Sensitive Hashing , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Michal Irani,et al.  Internal statistics of a single natural image , 2011, CVPR 2011.

[36]  Luc Van Gool,et al.  Transforming Image Completion , 2011, BMVC.

[37]  Jian Sun,et al.  Statistics of Patch Offsets for Image Completion , 2012, ECCV.

[38]  Steven M. Drucker,et al.  Quality prediction for image completion , 2012, ACM Trans. Graph..

[39]  Jian Sun,et al.  Computing nearest-neighbor fields via Propagation-Assisted KD-Trees , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Vladimir G. Kim,et al.  Symmetry-guided texture synthesis and manipulation , 2012, TOGS.

[41]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[42]  Ying Wu,et al.  Large Displacement Optical Flow from Nearest Neighbor Fields , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Yinda Zhang,et al.  FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.