Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.

[1]  Yücel Altunbasak,et al.  Color plane interpolation using alternating projections , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[2]  Tony F. Chan,et al.  The digital TV filter and nonlinear denoising , 2001, IEEE Trans. Image Process..

[3]  Michael Elad,et al.  Super-resolution reconstruction of continuous image sequences , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[4]  Nirmal K. Bose,et al.  Mathematical analysis of super-resolution methodology , 2003, IEEE Signal Process. Mag..

[5]  Michael Elad,et al.  Super-Resolution Reconstruction of Image Sequences , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Yücel Altunbasak,et al.  Super-resolution still and video reconstruction from MPEG-coded video , 2002, IEEE Trans. Circuits Syst. Video Technol..

[7]  Yacov Hel-Or,et al.  Demosaicing of Color Images Using Steerable Wavelets , 2002 .

[8]  Robert L. Stevenson,et al.  Super-resolution from image sequences-a review , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[9]  Michael Elad,et al.  Multiframe demosaicing and super-resolution of color images , 2006, IEEE Transactions on Image Processing.

[10]  Peyman Milanfar,et al.  A computationally efficient superresolution image reconstruction algorithm , 2001, IEEE Trans. Image Process..

[11]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[12]  Michael Elad,et al.  On the origin of the bilateral filter and ways to improve it , 2002, IEEE Trans. Image Process..

[13]  Alan C. Bovik,et al.  Handbook of Image and Video Processing (Communications, Networking and Multimedia) , 2005 .

[14]  Michael Elad,et al.  Superresolution restoration of an image sequence: adaptive filtering approach , 1999, IEEE Trans. Image Process..

[15]  Aggelos K. Katsaggelos,et al.  Resolution enhancement of monochrome and color video using motion compensation , 2001, IEEE Trans. Image Process..

[16]  Shmuel Peleg,et al.  Efficient super-resolution and applications to mosaics , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[17]  Andrew Harvey,et al.  Forecasting, Structural Time Series Models and the Kalman Filter , 1990 .

[18]  William K. Pratt,et al.  Digital Image Processing: PIKS Inside , 2001 .

[19]  Masahito Yamada,et al.  Structural Time Series Models and the Kalman Filter , 1989 .

[20]  Masatoshi Okutomi,et al.  Direct super-resolution and registration using raw CFA images , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[21]  C. Striebel,et al.  On the maximum likelihood estimates for linear dynamic systems , 1965 .

[22]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[23]  Sabine Süsstrunk,et al.  Color Demosaicing by Estimating Luminance and Opponent Chromatic Signals in the Fourier Domain , 2002, CIC.

[24]  Steven Kay,et al.  Fundamentals Of Statistical Signal Processing , 2001 .

[25]  Ron Kimmel,et al.  Demosaicing: Image Reconstruction from Color CCD Samples , 1998, ECCV.

[26]  Polina Golland,et al.  Motion from Color , 1997, Comput. Vis. Image Underst..

[27]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[28]  Rafael C. González,et al.  Digital image processing, 3rd Edition , 2008 .

[29]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[30]  Aggelos K. Katsaggelos,et al.  Bayesian resolution enhancement of compressed video , 2004, IEEE Transactions on Image Processing.

[31]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[32]  Wesley E. Snyder,et al.  Demosaicking methods for Bayer color arrays , 2002, J. Electronic Imaging.

[33]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[34]  Shmuel Peleg,et al.  Multi-sensor super-resolution , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[35]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[36]  Michael Elad,et al.  A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur , 2001, IEEE Trans. Image Process..

[37]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[38]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[39]  Soo-Chang Pei,et al.  Effective color interpolation in CCD color filter arrays using signal correlation , 2003, IEEE Trans. Circuits Syst. Video Technol..

[40]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[41]  Avideh Zakhor,et al.  Resolution enhancement of color video sequences , 1999, IEEE Trans. Image Process..

[42]  Jerry D. Gibson,et al.  Handbook of Image and Video Processing , 2000 .

[43]  Michael Elad,et al.  Multiframe demosaicing and super-resolution from undersampled color images , 2004, IS&T/SPIE Electronic Imaging.

[44]  Daniel Keren,et al.  Restoring subsampled color images , 1999, Machine Vision and Applications.

[45]  Yuying Li,et al.  A computational algorithm for minimizing total variation in image restoration , 1996, IEEE Trans. Image Process..

[46]  Michael Elad,et al.  Fast and Robust Multi-Frame Super-Resolution , 2004, IEEE Transactions on Image Processing.

[47]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[48]  Yuan Jian.  A Fast Super-resolution Reconstruction Algorithm , 2006 .