Geometrically Consistent Stereoscopic Image Editing Using Patch-Based Synthesis

This paper presents a patch-based synthesis framework for stereoscopic image editing. The core of the proposed method builds upon a patch-based optimization framework with two key contributions: First, we introduce a depth-dependent patch-pair similarity measure for distinguishing and better utilizing image contents with different depth structures. Second, a joint patch-pair search is proposed for properly handling the correlation between two views. The proposed method successfully overcomes two main challenges of editing stereoscopic 3D media: (1) maintaining the depth interpretation, and (2) providing controllability of the scene depth. The method offers patch-based solutions to a wide variety of stereoscopic image editing problems, including depth-guided texture synthesis, stereoscopic NPR, paint by depth, content adaptation, and 2D to 3D conversion. Several challenging cases are demonstrated to show the effectiveness of the proposed method. The results of user studies also show that the proposed method produces stereoscopic images with good stereoscopics and visual quality.

[1]  Yung-Yu Chuang,et al.  Scene warping: Layer-based stereoscopic image resizing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Alexander A. Sawchuk,et al.  Disparity manipulation for stereo images and video , 2008, Electronic Imaging.

[3]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[4]  Shai Avidan,et al.  Geometrically consistent stereo seam carving , 2011, 2011 International Conference on Computer Vision.

[5]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[6]  Matthias Zwicker,et al.  Stereoscopic 3D copy & paste , 2010, SIGGRAPH 2010.

[7]  Wen-Huang Cheng,et al.  Perspective-aware warping for seamless stereoscopic image cloning , 2012, ACM Trans. Graph..

[8]  Harry Shum,et al.  Image completion with structure propagation , 2005, ACM Trans. Graph..

[9]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[10]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Hailin Jin,et al.  Stereo matching with nonparametric smoothness priors in feature space , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[13]  Nipun Kwatra,et al.  Texture optimization for example-based synthesis , 2005, ACM Trans. Graph..

[14]  Yung-Yu Chuang,et al.  Content-Aware Display Adaptation and Interactive Editing for Stereoscopic Images , 2011, IEEE Transactions on Multimedia.

[15]  Eli Shechtman,et al.  Image melding , 2012, ACM Trans. Graph..

[16]  Wu-chi Feng,et al.  Enabling warping on stereoscopic images , 2012, ACM Trans. Graph..

[17]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[18]  M. Gross,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[19]  Markus H. Gross,et al.  StereoBrush: interactive 2D to 3D conversion using discontinuous warps , 2011, SBIM '11.

[20]  Eli Shechtman,et al.  Space-Time Completion of Video , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Sylvain Lefebvre,et al.  State of the Art in Example-based Texture Synthesis , 2009, Eurographics.

[22]  Michael F. Cohen,et al.  A Viewer-Centric Editor for 3D Movies , 2011, IEEE Computer Graphics and Applications.

[23]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[24]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[25]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[26]  Gabriel J. Brostow,et al.  Learning to find occlusion regions , 2011, CVPR 2011.

[27]  Baining Guo,et al.  Real-time texture synthesis by patch-based sampling , 2001, TOGS.

[28]  Brian L. Price,et al.  PatchMatch-Based Content Completion of Stereo Image Pairs , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[29]  Shai Avidan,et al.  Stereo Seam Carving a Geometrically Consistent Approach , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Craig S. Kaplan,et al.  Consistent stylization and painterly rendering of stereoscopic 3D images , 2012, NPAR '12.

[31]  Ruigang Yang,et al.  Stereoscopic inpainting: Joint color and depth completion from stereo images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Luc Van Gool,et al.  Scene Carving: Scene Consistent Image Retargeting , 2010, ECCV.