Invertible motion blur in video

We show that motion blur in successive video frames is invertible even if the point-spread function (PSF) due to motion smear in a single photo is non-invertible. Blurred photos exhibit nulls (zeros) in the frequency transform of the PSF, leading to an ill-posed deconvolution. Hardware solutions to avoid this require specialized devices such as the coded exposure camera or accelerating sensor motion. We employ ordinary video cameras and introduce the notion of null-filling along with joint-invertibility of multiple blur-functions. The key idea is to record the same object with varying PSFs, so that the nulls in the frequency component of one frame can be filled by other frames. The combined frequency transform becomes null-free, making deblurring well-posed. We achieve jointly-invertible blur simply by changing the exposure time of successive frames. We address the problem of automatic deblurring of objects moving with constant velocity by solving the four critical components: preservation of all spatial frequencies, segmentation of moving parts, motion estimation of moving parts, and non-degradation of the static parts of the scene. We demonstrate several challenging cases of object motion blur including textured backgrounds and partial occluders.

[1]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[2]  L. Lucy An iterative technique for the rectification of observed distributions , 1974 .

[3]  C. A. Berenstein,et al.  Exact deconvolution for multiple convolution operators-an overview, plus performance characterizations for imaging sensors , 1990, Proc. IEEE.

[4]  W. Cathey,et al.  Extended depth of field through wave-front coding. , 1995, Applied optics.

[5]  Worthy N. Martin,et al.  Image Motion Estimation From Motion Smear-A New Computational Model , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  P. Jansson Deconvolution of images and spectra (2nd ed.) , 1996 .

[7]  Andrew Blake,et al.  Motion Deblurring and Super-resolution from an Image Sequence , 1996, ECCV.

[8]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[9]  Yaron Caspi,et al.  Increasing Space-Time Resolution in Video , 2002, ECCV.

[10]  Shree K. Nayar,et al.  High Dynamic Range from Multiple Images: Which Exposures to Combine?∗ , 2003 .

[11]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[12]  Shree K. Nayar,et al.  Motion-based motion deblurring , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[14]  Shmuel Peleg,et al.  Two motion-blurred images are better than one , 2005, Pattern Recognit. Lett..

[15]  Ramesh Raskar,et al.  Coded exposure photography: motion deblurring using fluttered shutter , 2006, SIGGRAPH 2006.

[16]  William T. Freeman,et al.  Removing camera shake from a single photograph , 2006, SIGGRAPH 2006.

[17]  Ramesh Raskar,et al.  Resolving Objects at Higher Resolution from a Single Motion-blurred Image , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Jiaya Jia,et al.  Single Image Motion Deblurring Using Transparency , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Long Quan,et al.  Image deblurring with blurred/noisy image pairs , 2007, SIGGRAPH 2007.

[20]  Yasuyuki Matsushita,et al.  Removing Non-Uniform Motion Blur from Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[21]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, SIGGRAPH 2007.

[22]  Anne Sullivan,et al.  Synthetic Shutter Speed Imaging , 2007, Comput. Graph. Forum.

[23]  Jia Chen,et al.  Robust dual motion deblurring , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Jiaya Jia,et al.  High-quality motion deblurring from a single image , 2008, SIGGRAPH 2008.

[25]  Frédo Durand,et al.  Motion-invariant photography , 2008, SIGGRAPH 2008.

[26]  Jian Sun,et al.  Progressive inter-scale and intra-scale non-blind image deconvolution , 2008, SIGGRAPH 2008.

[27]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[28]  Richard Szeliski,et al.  PSF estimation using sharp edge prediction , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Shree K. Nayar,et al.  Flexible Depth of Field Photography , 2008, ECCV.

[30]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[31]  Ying Wu,et al.  Motion from blur , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Stephen Lin,et al.  Image/video deblurring using a hybrid camera , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Anita Sellent,et al.  Calculating Motion Fields from Images with Two Different Exposure Times , 2008 .