Generalizing the Nonlocal-Means to Super-Resolution Reconstruction

Super-resolution reconstruction proposes a fusion of several low-quality images into one higher quality result with better optical resolution. Classic super-resolution techniques strongly rely on the availability of accurate motion estimation for this fusion task. When the motion is estimated inaccurately, as often happens for nonglobal motion fields, annoying artifacts appear in the super-resolved outcome. Encouraged by recent developments on the video denoising problem, where state-of-the-art algorithms are formed with no explicit motion estimation, we seek a super-resolution algorithm of similar nature that will allow processing sequences with general motion patterns. In this paper, we base our solution on the Nonlocal-Means (NLM) algorithm. We show how this denoising method is generalized to become a relatively simple super-resolution algorithm with no explicit motion estimation. Results on several test movies show that the proposed method is very successful in providing super-resolution on general sequences.

[1]  D. Luenberger Optimization by Vector Space Methods , 1968 .

[2]  Thomas S. Huang,et al.  Multiframe image restoration and registration , 1984 .

[3]  Shmuel Peleg,et al.  Improving image resolution using subpixel motion , 1987, Pattern Recognit. Lett..

[4]  R. Mersereau,et al.  Iterative methods for image deblurring , 1990 .

[5]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[6]  George Wolberg,et al.  Digital image warping , 1990 .

[7]  Ken Turkowski,et al.  Filters for common resampling tasks , 1990 .

[8]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[9]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[10]  Daniel Gross,et al.  Improved resolution from subpixel shifted pictures , 1992, CVGIP Graph. Model. Image Process..

[11]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[12]  A. Murat Tekalp,et al.  Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time , 1997, IEEE Trans. Image Process..

[13]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[14]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[15]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[16]  Avideh Zakhor,et al.  Resolution enhancement of color video sequences , 1999, IEEE Trans. Image Process..

[17]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[18]  Shmuel Peleg,et al.  Efficient super-resolution and applications to mosaics , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[19]  Aggelos K. Katsaggelos,et al.  Resolution enhancement of monochrome and color video using motion compensation , 2001, IEEE Trans. Image Process..

[20]  Peyman Milanfar,et al.  A computationally efficient superresolution image reconstruction algorithm , 2001, IEEE Trans. Image Process..

[21]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[22]  Michael Elad,et al.  A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur , 2001, IEEE Trans. Image Process..

[23]  Michael Elad,et al.  On the origin of the bilateral filter and ways to improve it , 2002, IEEE Trans. Image Process..

[24]  Michael Elad,et al.  On the bilateral filter and ways to improve it , 2002 .

[25]  Yücel Altunbasak,et al.  Super-resolution still and video reconstruction from MPEG-coded video , 2002, IEEE Trans. Circuits Syst. Video Technol..

[26]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[27]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[28]  Michael Elad,et al.  Robust shift and add approach to superresolution , 2003, SPIE Optics + Photonics.

[29]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[30]  Zhouchen Lin,et al.  Response to the Comments on "Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation' , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Masatoshi Okutomi,et al.  Direct super-resolution and registration using raw CFA images , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[32]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[33]  Guillermo Sapiro,et al.  Fast image and video denoising via nonlocal means of similar neighborhoods , 2005, IEEE Signal Processing Letters.

[34]  Jaakko Astola,et al.  A spatially adaptive nonparametric regression image deblurring , 2005, IEEE Transactions on Image Processing.

[35]  T. R. Gosnell,et al.  Digital Image Reconstruction: Deblurring and Denoising , 2005 .

[36]  Gregory T. Clement,et al.  Superresolution ultrasound imaging using back-projected reconstruction. , 2005, The Journal of the Acoustical Society of America.

[37]  Jean-Michel Morel,et al.  Denoising image sequences does not require motion estimation , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[38]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[39]  José M. Bioucas-Dias,et al.  Bayesian wavelet-based image deconvolution: a GEM algorithm exploiting a class of heavy-tailed priors , 2006, IEEE Transactions on Image Processing.

[40]  Sabine Süsstrunk,et al.  A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution , 2006, EURASIP J. Adv. Signal Process..

[41]  Kostadin Dabov,et al.  MOVING-WINDOW VARYING SIZE 3D TRANSFORM-BASED VIDEO DENOISING , 2006 .

[42]  J. Nagy,et al.  Numerical methods for coupled super-resolution , 2006 .

[43]  Stanley J. Reeves,et al.  Efficient Huber-Markov Edge-Preserving Image Restoration , 2006, IEEE Transactions on Image Processing.

[44]  Liangpei Zhang,et al.  A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution , 2007, IEEE Transactions on Image Processing.

[45]  Peyman Milanfar,et al.  Kernel Regression for Image Processing and Reconstruction , 2007, IEEE Transactions on Image Processing.

[46]  Guy Gilboa,et al.  Nonlocal Linear Image Regularization and Supervised Segmentation , 2007, Multiscale Model. Simul..

[47]  Patrick Bouthemy,et al.  Space-Time Adaptation for Patch-Based Image Sequence Restoration , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Michael Elad,et al.  Image Sequence Denoising via Sparse and Redundant Representations , 2009, IEEE Transactions on Image Processing.

[49]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .