Super-resolution from internet-scale scene matching

In this paper, we present a highly data-driven approach to the task of single image super-resolution. Super-resolution is a challenging problem due to its massively under-constrained nature - for any low-resolution input there are numerous high-resolution possibilities. Our key observation is that, even with extremely low-res input images, we can use global scene descriptors and Internet-scale image databases to find similar scenes which provide ideal example textures to constrain the image upsampling problem. We quantitatively show that the statistics of scene matches are more predictive than internal image statistics for the super-resolution task. Finally, we build on recent patch-based texture transfer techniques to hallucinate texture detail and compare our super-resolution with other recent methods.

[1]  Wojciech Matusik,et al.  CG2Real: Improving the Realism of Computer Generated Images Using a Large Collection of Photographs , 2011, IEEE Transactions on Visualization and Computer Graphics.

[2]  Wojciech Matusik,et al.  Image restoration using online photo collections , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[3]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[5]  Stephen Lin,et al.  Super resolution using edge prior and single image detail synthesis , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Michael J. Black,et al.  Fields of Experts: a framework for learning image priors , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[7]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[8]  Alexei A. Efros,et al.  Segmenting Scenes by Matching Image Composites , 2009, NIPS.

[9]  Michal Irani,et al.  Internal statistics of a single natural image , 2011, CVPR 2011.

[10]  Raanan Fattal,et al.  Image upsampling via imposed edge statistics , 2007, ACM Trans. Graph..

[11]  Raanan Fattal,et al.  Image upsampling via texture hallucination , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[12]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[13]  Hans-Peter Seidel,et al.  Photo zoom: high resolution from unordered image collections , 2010, SIGGRAPH '10.

[14]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  Alexei A. Efros,et al.  IM2GPS: estimating geographic information from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Richard Szeliski,et al.  Image deblurring and denoising using color priors , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  William T. Freeman,et al.  What makes a good model of natural images? , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[19]  Antonio Torralba,et al.  Building the gist of a scene: the role of global image features in recognition. , 2006, Progress in brain research.

[20]  Jason Lawrence,et al.  Building and Using a Database of One Trillion Natural-Image Patches , 2011, IEEE Computer Graphics and Applications.

[21]  Richard Szeliski,et al.  Video and Image Bayesian Demosaicing with a Two Color Image Prior , 2006, ECCV.

[22]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[23]  Richard Szeliski,et al.  A content-aware image prior , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Harry Shum,et al.  Face Hallucination: Theory and Practice , 2007, International Journal of Computer Vision.

[26]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[27]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[28]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[29]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Michael J. Black,et al.  Steerable Random Fields , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[31]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[32]  Edward H. Adelson,et al.  Exploring features in a Bayesian framework for material recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Krista A. Ehinger,et al.  SUN database: Large-scale scene recognition from abbey to zoo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Yihong Gong,et al.  Visual‐Quality Optimizing Super Resolution , 2009, Comput. Graph. Forum.

[35]  Jiejie Zhu,et al.  Context-constrained hallucination for image super-resolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Song-Chun Zhu,et al.  Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling , 1998, International Journal of Computer Vision.

[37]  Michael Isard,et al.  Lost in quantization: Improving particular object retrieval in large scale image databases , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.