Image upsampling via texture hallucination

Image upsampling is a common yet challenging task, since it is severely underconstrained. While considerable progress was made in preserving the sharpness of salient edges, current methods fail to reproduce the fine detail typically present in the textured regions bounded by these edges, resulting in unrealistic appearance. In this paper we address this fundamental shortcoming by integrating higher-level image analysis and custom low-level image synthesis. Our approach extends and refines the patch-based image model of Freeman et al. [10] and interprets the image as a tiling of distinct textures, each of which is matched to an example in a database of relevant textures. The matching is not done at the patch level, but rather collectively, over entire segments. Following this model fitting stage, which requires some user guidance, a higher-resolution image is synthesized using a hybrid approach that incorporates principles from example-based texture synthesis. We show that for images that comply with our model, our method is able to reintroduce consistent fine-scale detail, resulting in enhanced appearance textured regions.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  James R. Bergen,et al.  Pyramid-based texture analysis/synthesis , 1995, Proceedings., International Conference on Image Processing.

[3]  William T. Freeman,et al.  Learning to Estimate Scenes from Images , 1998, NIPS.

[4]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Michael Unser,et al.  Image interpolation and resampling , 2000 .

[6]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[7]  Takeo Kanade,et al.  Hallucinating faces , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  M. Orchard,et al.  New edge-directed interpolation , 2001, IEEE Trans. Image Process..

[9]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[10]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[11]  S. Osher,et al.  IMAGE DECOMPOSITION AND RESTORATION USING TOTAL VARIATION MINIMIZATION AND THE H−1 NORM∗ , 2002 .

[12]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[13]  Stanley Osher,et al.  Image Decomposition and Restoration Using Total Variation Minimization and the H1 , 2003, Multiscale Model. Simul..

[14]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[15]  Brendan J. Frey,et al.  Epitomic analysis of appearance and shape , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[16]  Nanning Zheng,et al.  Image hallucination with primal sketch priors , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[17]  William T. Freeman,et al.  Efficient graphical models for processing images , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[18]  Jitendra Malik,et al.  Contour and Texture Analysis for Image Segmentation , 2001, International Journal of Computer Vision.

[19]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[20]  Philip J. Willis,et al.  Image Interpolation by Pixel‐Level Data‐Dependent Triangulation , 2004, Comput. Graph. Forum.

[21]  Nipun Kwatra,et al.  Texture optimization for example-based synthesis , 2005, ACM Trans. Graph..

[22]  Eric Dubois,et al.  Image up-sampling using total-variation regularization with a new observation model , 2005, IEEE Transactions on Image Processing.

[23]  Harry Shum,et al.  Face Hallucination: Theory and Practice , 2007, International Journal of Computer Vision.

[24]  Maneesh Agrawala,et al.  Using Photographs to Enhance Videos of a Static Scene , 2007, Rendering Techniques.

[25]  Antonio Torralba,et al.  Object Recognition by Scene Alignment , 2007, NIPS.

[26]  Raanan Fattal,et al.  Image upsampling via imposed edge statistics , 2007, ACM Trans. Graph..

[27]  Mehran Ebrahimi,et al.  Solving the Inverse Problem of Image Zooming Using "Self-Examples" , 2007, ICIAR.

[28]  H. Shum,et al.  Image super-resolution using gradient profile prior , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Bobby Bodenheimer,et al.  Synthesis and evaluation of linear motion transitions , 2008, TOGS.

[30]  Eitan Grinspun,et al.  Multiscale texture synthesis , 2008, ACM Trans. Graph..

[31]  Chi-Keung Tang,et al.  Fast image/video upsampling , 2008, SIGGRAPH Asia '08.

[32]  Maneesh Agrawala,et al.  Edge-based image coarsening , 2009, TOGS.

[33]  Sylvain Lefebvre,et al.  State of the Art in Example-based Texture Synthesis , 2009, Eurographics.

[34]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[35]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[36]  Ronen Basri,et al.  Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  N. H. Beebe.  Multiscale Modeling & Simulation , 2022 .