Data-driven multichannel superresolution with application to video sequences

A method is proposed for superresolving multichannel data with application to video sequences. Based on a generalization of Papoulis’ sampling theorem, nonuniform samples of multiple channels are merged to generate high-resolution data. To overcome sampling ill posedness in the presence of noise, image frames are projected from standard orthonormal bases onto optimal Riesz bases defined by channel point spread functions (PSF’s). The method is therefore designed to perform under practical conditions of noise and other degradations. Unlike existing methods, where empirical models such as Gaussian, sinc, etc., are commonly used for characterizing channel PSF’s, the PSF’s are assumed unknown and possibly different and hence are blindly estimated from the observed data. The estimated PSF’s are then used to construct biorthogonal projection filters for the superresolution algorithm. This approach gives rise to a closed-form solution leading to a high-speed algorithm. The method has been tested and verified on PREDATOR video sequences (PREDATOR data are airborne video sequences of the Defense Advanced Research Projects Agency obtained by unmanned aircrafts).

[1]  A. G. Greenhill,et al.  A Treatise on Bessel Functions and their Applications to Physics , 1895, Nature.

[2]  Andrew Calway,et al.  Proceedings of the IEEE International Conference on Image Processing , 1996 .

[3]  R. Gerchberg.  Super-resolution through Error Energy Reduction , 1974 .

[4]  Ramakrishna Kakarala,et al.  High resolution image reconstruction by simulated annealing , 1994 .

[5]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[6]  Josiane Zerubia,et al.  Subpixel image registration by estimating the polyphase decomposition of cross power spectrum , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[8]  A. Murat Tekalp,et al.  Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time , 1997, IEEE Trans. Image Process..

[9]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[10]  Stéphane Mallat,et al.  Best basis algorithm for signal enhancement , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[11]  C. Morandi,et al.  Registration of Translated and Rotated Images Using Finite Fourier Transforms , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[13]  Bobby R. Hunt,et al.  Description of a Poisson Imagery Super Resolution Algorithm , 1992 .

[14]  D. Slepian.  Prolate spheroidal wave functions, fourier analysis, and uncertainty — V: the discrete case , 1978, The Bell System Technical Journal.

[15]  Bobby R. Hunt,et al.  Imagery super-resolution: emerging prospects , 1991, Optics & Photonics.

[16]  Kiyoharu Aizawa,et al.  A scheme for acquiring very high resolution images using multiple cameras , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[17]  D. Slepian.  Prolate spheroidal wave functions, Fourier analysis and uncertainty — IV: Extensions to many dimensions; generalized prolate spheroidal functions , 1964 .

[18]  Ronald R. Coifman,et al.  Entropy-based algorithms for best basis selection , 1992, IEEE Trans. Inf. Theory.

[19]  J. L. Harris,et al.  Diffraction and Resolving Power , 1964 .

[20]  Ming-Chao Chiang,et al.  Local blur estimation and super-resolution , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  C. A. Berenstein,et al.  Exact deconvolution for multiple convolution operators-an overview, plus performance characterizations for imaging sensors , 1990, Proc. IEEE.

[22]  R.N. Bracewell,et al.  Signal analysis , 1978, Proceedings of the IEEE.

[23]  Helly.  Fourier transforms in the complex domain , 1936 .

[24]  A. Papoulis,et al.  Generalized sampling expansion , 1977 .

[25]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[26]  H. Pollak,et al.  Prolate spheroidal wave functions, fourier analysis and uncertainty — III: The dimension of the space of essentially time- and band-limited signals , 1962 .

[27]  S.D. Halford,et al.  Adaptive blind channel identification of FIR channels for Viterbi decoding , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[28]  Andrew J. Patti,et al.  High resolution standards conversion of low resolution video , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[29]  Michael Werman,et al.  Reconstruction of high resolution 3D visual information , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[30]  D. Slepian,et al.  Prolate spheroidal wave functions, fourier analysis and uncertainty — II , 1961 .

[31]  Daniel Gross,et al.  Improved resolution from subpixel shifted pictures , 1992, CVGIP Graph. Model. Image Process..

[32]  Josiane Zerubia,et al.  3D super-resolution using generalized sampling expansion , 1995, Proceedings., International Conference on Image Processing.

[33]  A. Papoulis.  A new algorithm in spectral analysis and band-limited extrapolation. , 1975 .

[34]  D. Donoho,et al.  Basis pursuit , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[35]  N. Wiener,et al.  Fourier Transforms in the Complex Domain , 1934 .

[36]  Wen-Yu Su,et al.  Recursive high-resolution reconstruction of blurred multiframe images , 1993, IEEE Trans. Image Process..

[37]  R. Young,et al.  An introduction to nonharmonic Fourier series , 1980 .

[38]  M. A. Fiddy,et al.  Phase ambiguities and the zeros of multidimensional band-limited functions , 1985 .

[39]  A. Murat Tekalp,et al.  High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[40]  Robert W. Heath,et al.  Blurs and perfect image restoration , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.