Extraction of high-resolution frames from video sequences

The human visual system appears to be capable of temporally integrating information in a video sequence in such a way that the perceived spatial resolution of a sequence appears much higher than the spatial resolution of an individual frame. While the mechanisms in the human visual system that do this are unknown, the effect is not too surprising given that temporally adjacent frames in a video sequence contain slightly different, but unique, information. This paper addresses the use of both the spatial and temporal information present in a short image sequence to create a single high-resolution video frame. A novel observation model based on motion compensated subsampling is proposed for a video sequence. Since the reconstruction problem is ill-posed, Bayesian restoration with a discontinuity-preserving prior image model is used to extract a high-resolution video still given a short low-resolution sequence. Estimates computed from a low-resolution image sequence containing a subpixel camera pan show dramatic visual and quantitative improvements over bilinear, cubic B-spline, and Bayesian single frame interpolations. Visual and quantitative improvements are also shown for an image sequence containing objects moving with independent trajectories. Finally, the video frame extraction algorithm is used for the motion-compensated scan conversion of interlaced video data, with a visual comparison to the resolution enhancement obtained from progressively scanned frames.

[1]  Peter Cheeseman,et al.  Super-Resolved Surface Reconstruction from Multiple Images , 1996 .

[2]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[3]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[4]  Robert Goutte,et al.  Image resolution enhancement using subpixel camera displacement , 1992, Signal Process..

[5]  Michael Unser,et al.  Fast B-spline Transforms for Continuous Image Representation and Interpolation , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Hsieh Hou,et al.  Cubic splines for image interpolation and digital filtering , 1978 .

[7]  P. Pirsch,et al.  Advances in picture coding , 1985, Proceedings of the IEEE.

[8]  A. Murat Tekalp,et al.  High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[9]  Michael Unser,et al.  Recursive Regularization Filters: Design, Properties, and Applications , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  H Stark,et al.  High-resolution image recovery from image-plane arrays, using convex projections. , 1989, Journal of the Optical Society of America. A, Optics and image science.

[11]  Andrew J. Patti,et al.  High resolution standards conversion of low resolution video , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[12]  Edward J. Delp,et al.  Discontinuity preserving regularization of inverse visual problems , 1994, IEEE Trans. Syst. Man Cybern..

[13]  Rui J. P. de Figueiredo,et al.  A unified approach to optimal image interpolation problems based on linear partial differential equation models , 1993, IEEE Trans. Image Process..

[14]  M. Hötter,et al.  Image segmentation based on object oriented mapping parameter estimation , 1988 .

[15]  A. Murat Tekalp,et al.  High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur , 1992, Proceedings of 1st International Conference on Image Processing.

[16]  Michael T. Orchard,et al.  A Comparison of Techniques for Estimating Block Motion in Image Sequence Coding , 1989, Other Conferences.

[17]  Robert L. Stevenson,et al.  A Bayesian approach to image expansion for improved definitio , 1994, IEEE Trans. Image Process..

[18]  J. A. Parker,et al.  Comparison of Interpolating Methods for Image Resampling , 1983, IEEE Transactions on Medical Imaging.

[19]  Anil K. Jain.  Fundamentals of Digital Image Processing , 2018, Control of Color Imaging Systems.

[20]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[21]  Robert L. Stevenson,et al.  Stochastic modeling and estimation of multispectral image data , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[22]  Gerard de Haan,et al.  Sub-pixel motion estimation with 3-D recursive search block-matching , 1994, Signal Process. Image Commun..

[23]  Rui J. P. de Figueiredo,et al.  Two-dimensional interpolation by generalized spline filters based on partial differential equation image models , 1985, IEEE Trans. Acoust. Speech Signal Process..

[24]  Eric Walowit,et al.  An edge-restricted spatial interpolation algorithm , 1992, J. Electronic Imaging.

[25]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[26]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[27]  D. Shulman,et al.  Regularization of discontinuous flow fields , 1989, [1989] Proceedings. Workshop on Visual Motion.

[28]  M. Bierling,et al.  Displacement Estimation By Hierarchical Blockmatching , 1988, Other Conferences.

[29]  R. Keys.  Cubic convolution interpolation for digital image processing , 1981 .

[30]  Anastasios N. Venetsanopoulos,et al.  Image interpolation based on variational principles , 1991, Signal Process..