Algorithms for super-resolution of images and videos based on learning methods. (Algorithmes de super-résolution d'images et de vidéos basés sur des méthodes d'apprentissage)

With super-resolution (SR) we refer to a class of techniques that enhance the spatial resolution of images and videos. SR algorithms can be of two kinds: multi-frame methods, where multiple low-resolution images are aggregated to form a unique high-resolution image, and single-image methods, that aim at upscaling a single image. This thesis focuses on developing theory and algorithms for the single-image SR problem. In particular, we adopt the so called example-based approach, where the output image is estimated with machine learning techniques, by using the information contained in a dictionary of image “examples”. The examples consist in image patches, which are either extracted from external images or derived from the input image itself. For both kinds of dictionary, we design novel SR algorithms, with new upscaling and dictionary construction procedures, and compare them to state-of-the-art methods. The results achieved are shown to be very competitive both in terms of visual quality of the super-resolved images and computational complexity. We then apply our designed algorithms to the video upscaling case, where the goal is to enlarge the resolution of an entire video sequence. The algorithms, opportunely adapted to deal with this case, are also analyzed in the coding context. The analysis conducted shows that, in specific cases, SR can also be an effective tool for video compression, thus opening new interesting perspectives.

[1]  Chih-Yuan Yang,et al.  Exploiting Self-similarities for Single Frame Super-Resolution , 2010, ACCV.

[2]  Kaggere V Suresh,et al.  Robust and computationally efficient superresolution algorithm. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Hugh G. Lewis,et al.  Super-resolution target identification from remotely sensed images using a Hopfield neural network , 2001, IEEE Trans. Geosci. Remote. Sens..

[4]  Lei Zhang,et al.  Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization , 2010, IEEE Transactions on Image Processing.

[5]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[6]  R. Casey,et al.  Advances in Pattern Recognition , 1971 .

[7]  Nirmal K. Bose,et al.  Simultaneous noise filtering and super-resolution with second-generation wavelets , 2005, IEEE Signal Processing Letters.

[8]  Moon Gi Kang,et al.  DCT-based regularized algorithm for high-resolution image reconstruction , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[9]  Soheil Darabi,et al.  Compressive image super-resolution , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[10]  Wan-Chi Siu,et al.  Single image super-resolution using Gaussian process regression , 2011, CVPR 2011.

[11]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Michal Irani,et al.  Super resolution from image sequences , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[13]  Nikolas P. Galatsanos,et al.  Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images , 2006, IEEE Transactions on Image Processing.

[14]  Jorge Núñez,et al.  Super-Resolution of Remotely Sensed Images With Variable-Pixel Linear Reconstruction , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[16]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[17]  Yi Wang,et al.  Super-resolution mosaicking of UAV surveillance video , 2008, 2008 15th IEEE International Conference on Image Processing.

[18]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[19]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[20]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[22]  Raanan Fattal,et al.  Image upsampling via imposed edge statistics , 2007, ACM Trans. Graph..

[23]  Daniel Gross,et al.  Improved resolution from subpixel shifted pictures , 1992, CVGIP Graph. Model. Image Process..

[24]  Nirmal K. Bose,et al.  Superresolution and noise filtering using moving least squares , 2006, IEEE Transactions on Image Processing.

[25]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[26]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[27]  R. Keys Cubic convolution interpolation for digital image processing , 1981 .

[28]  Eric Dubois,et al.  Image up-sampling using total-variation regularization with a new observation model , 2005, IEEE Transactions on Image Processing.

[29]  Lawrence K. Saul,et al.  Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifold , 2003, J. Mach. Learn. Res..

[30]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[31]  Meriç Özcan,et al.  Moving target detection using super‐resolution algorithms with an ultra wideband radar , 2010, Int. J. Imaging Syst. Technol..

[32]  Hua Huang,et al.  Neighbor embedding based super-resolution algorithm through edge detection and feature selection , 2009, Pattern Recognit. Lett..

[33]  John H. Reif,et al.  Super-Resolution Video Analysis for Forensic Investigations , 2007, IFIP Int. Conf. Digital Forensics.

[34]  Cornelia Fermüller,et al.  Robust Wavelet-Based Super-Resolution Reconstruction: Theory and Algorithm , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Kai-Kuang Ma,et al.  Adaptive rood pattern search for fast block-matching motion estimation , 2002, IEEE Trans. Image Process..

[36]  Shiguang Shan,et al.  Locality preserving constraints for super-resolution with neighbor embedding , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[37]  Chris H. Q. Ding,et al.  Convex and Semi-Nonnegative Matrix Factorizations , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[39]  Chi-Keung Tang,et al.  Fast image/video upsampling , 2008, SIGGRAPH Asia '08.

[40]  Byung Cheol Song,et al.  Fast Super-Resolution Algorithm Based on Dictionary Size Reduction Using k-Means Clustering , 2010 .

[41]  Kai-Kuang Ma,et al.  A survey on super-resolution imaging , 2011, Signal Image Video Process..

[42]  Xuelong Li,et al.  Image Super-Resolution With Sparse Neighbor Embedding , 2012, IEEE Transactions on Image Processing.

[43]  Franz Pernkopf,et al.  Sparse nonnegative matrix factorization using ℓ0-constraints , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[44]  Robert L. Stevenson,et al.  Super-resolution from image sequences-a review , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[45]  Aline Roumy,et al.  K-WEB: Nonnegative dictionary learning for sparse image representations , 2013, 2013 IEEE International Conference on Image Processing.

[46]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[47]  Nicu Sebe,et al.  Neighborhood issue in single-frame image super-resolution , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[48]  Majid Mirmehdi,et al.  An Introduction to Super-Resolution Text , 2007 .

[49]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[50]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[51]  Aline Roumy,et al.  Video super-resolution via sparse combinations of key-frame patches in a compression context , 2013, 2013 Picture Coding Symposium (PCS).

[52]  Mei Han,et al.  Soft Edge Smoothness Prior for Alpha Channel Super Resolution , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  A. Murat Tekalp,et al.  Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time , 1997, IEEE Trans. Image Process..

[54]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[55]  Lei Zhang,et al.  Sparse Representation Based Image Interpolation With Nonlocal Autoregressive Modeling , 2013, IEEE Transactions on Image Processing.

[56]  Michal Irani,et al.  Internal statistics of a single natural image , 2011, CVPR 2011.

[57]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[58]  Aggelos K. Katsaggelos,et al.  Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images , 1995, Proceedings., International Conference on Image Processing.

[59]  A. Papoulis,et al.  Generalized sampling expansion , 1977 .

[60]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[61]  K. Siddaraju,et al.  DIGITAL IMAGE RESTORATION , 2011 .

[62]  Said Esmail El-Khamy,et al.  Wavelet Fusion: a Tool to Break the Limits on LMMSE Image Super-Resolution , 2006, Int. J. Wavelets Multiresolution Inf. Process..

[63]  Aline Roumy,et al.  Compact and coherent dictionary construction for example-based super-resolution , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[64]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[65]  Motoyuki Sato,et al.  Super-resolution of coherent targets by a directional borehole radar , 2000, IEEE Trans. Geosci. Remote. Sens..

[66]  Hayit Greenspan,et al.  Super-Resolution in Medical Imaging , 2009, Comput. J..

[67]  Yihong Gong,et al.  Resolution enhancement based on learning the sparse association of image patches , 2010, Pattern Recognit. Lett..

[68]  Dit-Yan Yeung,et al.  Image Hallucination Using Neighbor Embedding over Visual Primitive Manifolds , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[69]  Michael Elad,et al.  K-SVD and its non-negative variant for dictionary design , 2005, SPIE Optics + Photonics.

[70]  Yu-Chiang Frank Wang,et al.  Learning context-aware sparse representation for single image super-resolution , 2011, 2011 18th IEEE International Conference on Image Processing.

[71]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[72]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[73]  Zhe L. Lin,et al.  Fast Image Super-Resolution Based on In-Place Example Regression , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[74]  Kai-Kuang Ma,et al.  Stochastic super-resolution image reconstruction , 2010, J. Vis. Commun. Image Represent..

[75]  Liangpei Zhang,et al.  A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution , 2007, IEEE Transactions on Image Processing.

[76]  Bryan C. Russell,et al.  Exploiting the sparse derivative prior for super-resolution , 2003 .

[77]  Xuelong Li,et al.  Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression , 2012, IEEE Transactions on Image Processing.

[78]  Mei Han,et al.  SoftCuts: A Soft Edge Smoothness Prior for Color Image Super-Resolution , 2009, IEEE Transactions on Image Processing.

[79]  Aline Roumy,et al.  Neighbor embedding based single-image super-resolution using Semi-Nonnegative Matrix Factorization , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[80]  Edson M. Hung,et al.  Video Super-Resolution Using Codebooks Derived From Key-Frames , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[81]  Aline Roumy,et al.  Super-resolution using neighbor embedding of back-projection residuals , 2013, 2013 18th International Conference on Digital Signal Processing (DSP).

[82]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[83]  Nanning Zheng,et al.  Image hallucination with primal sketch priors , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[84]  Xuelong Li,et al.  Single-image super-resolution via local learning , 2011, Int. J. Mach. Learn. Cybern..

[85]  Hui Ji,et al.  Robust wavelet-based super-resolution reconstruction: theory and algorithm. , 2009, IEEE transactions on pattern analysis and machine intelligence.

[86]  Nikolas P. Galatsanos,et al.  Maximum a Posteriori Video Super-Resolution Using a New Multichannel Image Prior , 2010, IEEE Transactions on Image Processing.

[87]  Thomas S. Huang,et al.  Image super-resolution as sparse representation of raw image patches , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[88]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[89]  Peyman Milanfar,et al.  An efficient wavelet-based algorithm for image superresolution , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).