How Useful Is Image Super-resolution to Other Vision Tasks?

Despite the great advances made on image superresolution (ISR) during the last years, the performance has solely been evaluated perceptually. Thus, it is still unclear how useful ISR is to other vision tasks in practice. In this paper, we present the first comprehensive study and analysis of the usefulness of ISR for other vision applications. In particular, five ISR methods are evaluated on four popular vision tasks, namely edge detection, semantic image labeling, digit recognition, and face detection. We show that applying ISR to input images of other vision systems does improve the performance when the input images are of low-resolution. This is because the features and algorithms of current vision systems are designed and optimized for images of normal resolution. We also demonstrate that the standard perceptual evaluation criteria, such as PSNR and SSIM, correlate quite well with the usefulness of ISR methods to other vision tasks, but cannot measure it very accurately. We hope this work will inspire the community to evaluate ISR methods also in real vision applications, and to deploy ISR as a preprocessing component for systems of other vision tasks if the input data are of relatively low-resolution.

[1]  Michael Unser,et al.  Image interpolation and resampling , 2000 .

[2]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[3]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[4]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[5]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[7]  Erik Learned-Miller,et al.  FDDB: A benchmark for face detection in unconstrained settings , 2010 .

[8]  Antonio Criminisi,et al.  TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation , 2006, ECCV.

[9]  Roberto Cipolla,et al.  Semantic texton forests for image categorization and segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Pablo H. Hennings-Yeomans,et al.  Simultaneous super-resolution and feature extraction for recognition of low-resolution faces , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Xiao-Yuan Jing,et al.  Super-resolution Person re-identification with semi-coupled low-rank discriminant dictionary learning , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Yücel Altunbasak,et al.  Eigenface-domain super-resolution for face recognition , 2003, IEEE Trans. Image Process..

[13]  Mei Han,et al.  Soft Edge Smoothness Prior for Alpha Channel Super Resolution , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Luc Van Gool,et al.  A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution , 2014, ACCV.

[15]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[16]  Luc Van Gool,et al.  Learning Domain Knowledge for Façade Labelling , 2012, ECCV.

[17]  Horst Bischof,et al.  Fast and accurate image upscaling with super-resolution forests , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[19]  Luc Van Gool,et al.  Jointly Optimized Regressors for Image Super‐resolution , 2015, Comput. Graph. Forum.

[20]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[21]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[22]  Edward H. Adelson,et al.  Crisp Boundary Detection Using Pointwise Mutual Information , 2014, ECCV.

[23]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yanning Zhang,et al.  Modeling deformable gradient compositions for single-image super-resolution , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Rachid Deriche,et al.  Vector-valued image regularization with PDEs: a common framework for different applications , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Bryan C. Russell,et al.  Exploiting the sparse derivative prior for super-resolution , 2003 .

[27]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[28]  Xuming He,et al.  Superpixel Graph Label Transfer with Learned Distance Metric , 2014, ECCV.

[29]  Luc Van Gool,et al.  Metric imitation by manifold transfer for efficient vision applications , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  C. Duchon.  Lanczos Filtering in One and Two Dimensions , 1979 .