A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution

Super resolution image reconstruction allows the recovery of a high-resolution (HR) image from several low-resolution images that are noisy, blurred, and down sampled. In this paper, we present a joint formulation for a complex super-resolution problem in which the scenes contain multiple independently moving objects. This formulation is built upon the maximum a posteriori (MAP) framework, which judiciously combines motion estimation, segmentation, and super resolution together. A cyclic coordinate descent optimization procedure is used to solve the MAP formulation, in which the motion fields, segmentation fields, and HR images are found in an alternate manner given the two others, respectively. Specifically, the gradient-based methods are employed to solve the HR image and motion fields, and an iterated conditional mode optimization method to obtain the segmentation fields. The proposed algorithm has been tested using a synthetic image sequence, the "Mobile and Calendar" sequence, and the original "Motorcycle and Car" sequence. The experiment results and error analyses verify the efficacy of this algorithm

[1]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[2]  Avideh Zakhor,et al.  Resolution enhancement of color video sequences , 1999, IEEE Trans. Image Process..

[3]  A. Murat Tekalp,et al.  Simultaneous motion estimation and segmentation , 1997, IEEE Trans. Image Process..

[4]  Aggelos K. Katsaggelos,et al.  High‐resolution images from compressed low‐resolution video: Motion estimation and observable pixels , 2004, Int. J. Imaging Syst. Technol..

[5]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[6]  Aggelos K. Katsaggelos,et al.  Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images , 1995, Proceedings., International Conference on Image Processing.

[7]  A. Murat Tekalp,et al.  High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[8]  Russell C. Hardie,et al.  High resolution image reconstruction from digital video with global and non-global scene motion , 1997, Proceedings of International Conference on Image Processing.

[9]  A. Murat Tekalp,et al.  An algorithm for simultaneous motion estimation and scene segmentation , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[10]  A. Murat Tekalp,et al.  Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time , 1997, IEEE Trans. Image Process..

[11]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[12]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[13]  P. Anandan,et al.  Accurate computation of optical flow by using layered motion representations , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[14]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[15]  Michael Elad,et al.  Multiframe demosaicing and super-resolution of color images , 2006, IEEE Transactions on Image Processing.

[16]  Petar M. Djuric,et al.  Unsupervised vector image segmentation by a tree structure-ICM algorithm , 1996, IEEE Trans. Medical Imaging.

[17]  Wen-Yu Su,et al.  Recursive high-resolution reconstruction of blurred multiframe images , 1993, IEEE Trans. Image Process..

[18]  A. Papoulis,et al.  Generalized sampling expansion , 1977 .

[19]  Michael Elad,et al.  Superresolution restoration of an image sequence: adaptive filtering approach , 1999, IEEE Trans. Image Process..

[20]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[21]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[22]  Peyman Milanfar,et al.  Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement , 2001, IEEE Trans. Image Process..

[23]  Nirmal K. Bose,et al.  Recursive implementation of total least squares algorithm for image reconstruction from noisy, undersampled multiframes , 1993, 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[24]  Daniel Gross,et al.  Improved resolution from subpixel shifted pictures , 1992, CVGIP Graph. Model. Image Process..

[25]  Ali Tabatabai,et al.  Motion Estimation Methods for Video Compression—A Review , 1998 .

[26]  Robert L. Stevenson,et al.  Spatial Resolution Enhancement of Low-Resolution Image Sequences A Comprehensive Review with Directions for Future Research , 1998 .

[27]  Moon Gi Kang,et al.  Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration , 2003, IEEE Trans. Image Process..

[28]  Thrasyvoulos N. Pappas,et al.  An Adaptive Clustering Algorithm For Image Segmentation , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[29]  Seunghyeon Rhee,et al.  Discrete cosine transform based regularized high-resolution image reconstruction algorithm , 1999 .

[30]  Aggelos K. Katsaggelos,et al.  High-resolution images from low-resolution compressed video , 2003, IEEE Signal Process. Mag..

[31]  A. Murat Tekalp,et al.  High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur , 1992, Proceedings of 1st International Conference on Image Processing.

[32]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[33]  Michal Irani,et al.  Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency , 1993, J. Vis. Commun. Image Represent..

[34]  Aggelos K. Katsaggelos,et al.  Reconstruction of a high-resolution image from multiple-degraded misregistered low-resolution images , 1994, Other Conferences.

[35]  H Stark,et al.  High-resolution image recovery from image-plane arrays, using convex projections. , 1989, Journal of the Optical Society of America. A, Optics and image science.

[36]  Nikolas P. Galatsanos,et al.  Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images , 2006, IEEE Transactions on Image Processing.

[37]  J. Nagy,et al.  Numerical methods for coupled super-resolution , 2006 .

[38]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[39]  A. Murat Tekalp,et al.  Robust, object-based high-resolution image reconstruction from low-resolution video , 1997, IEEE Trans. Image Process..

[40]  오승준.  [서평]「Digital Video Processing」 , 1996 .

[41]  J. Yen.  On Nonuniform Sampling of Bandwidth-Limited Signals , 1956 .

[42]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[43]  Aggelos K. Katsaggelos,et al.  Bayesian resolution enhancement of compressed video , 2004, IEEE Transactions on Image Processing.

[44]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[45]  M. Ng,et al.  Analysis of displacement errors in high-resolution image reconstruction with multisensors , 2002 .