Joint image and depth completion in shape-from-focus: taking a cue from parallax.

Shape-from-focus (SFF) uses a sequence of space-variantly defocused observations captured with relative motion between camera and scene. It assumes that there is no motion parallax in the frames. This is a restriction and constrains the working environment. Moreover, SFF cannot recover the structure information when there are missing data in the frames due to CCD sensor damage or unavoidable occlusions. The capability of filling-in plausible information in regions devoid of data is of critical importance in many applications. Images of 3D scenes captured by off-the-shelf cameras with relative motion commonly exhibit parallax-induced pixel motion. We demonstrate the interesting possibility of exploiting motion parallax cue in the images captured in SFF with a practical camera to jointly inpaint the focused image and depth map.

[1]  A N Rajagopalan,et al.  Extension of the shape from focus method for reconstruction of high-resolution images. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  Kiriakos N. Kutulakos,et al.  A Layer-Based Restoration Framework for Variable-Aperture Photography , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[3]  Ashish Raj,et al.  A graph cut algorithm for generalized image deconvolution , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[4]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[5]  Edmund Y. Lam,et al.  A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video , 2007, EURASIP J. Adv. Signal Process..

[6]  Vladimir Kolmogorov,et al.  Optimizing Binary MRFs via Extended Roof Duality , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Steven A. Shafer,et al.  What is the center of the image? , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Célia A. Zorzo Barcelos,et al.  Image restoration using digital inpainting and noise removal , 2007, Image Vis. Comput..

[9]  Shree K. Nayar,et al.  Telecentric Optics for Focus Analysis , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Cheng Lu,et al.  Entropy Minimization for Shadow Removal , 2009, International Journal of Computer Vision.

[11]  Jianhong Shen,et al.  Digital inpainting based on the Mumford–Shah–Euler image model , 2002, European Journal of Applied Mathematics.

[12]  Guillermo Sapiro,et al.  Inpainting surface holes , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[13]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Trevor Darrell,et al.  Pyramid based depth from focus , 1988, Proceedings CVPR '88: The Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Anil C. Kokaram,et al.  On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach , 2004, IEEE Transactions on Image Processing.

[16]  Dani Lischinski,et al.  The Shadow Meets the Mask: Pyramid‐Based Shadow Removal , 2008, Comput. Graph. Forum.

[17]  Shree K. Nayar,et al.  Shape from Focus , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Vladimir Kolmogorov,et al.  Minimizing Nonsubmodular Functions with Graph Cuts-A Review , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Stan Z. Li,et al.  Markov Random Field Modeling in Computer Vision , 1995, Computer Science Workbench.

[21]  Marc Levoy,et al.  Synthetic aperture confocal imaging , 2004, ACM Trans. Graph..

[22]  Marc Levoy,et al.  Reconstructing Occluded Surfaces Using Synthetic Apertures: Stereo, Focus and Robust Measures , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[23]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[24]  Guillermo Sapiro,et al.  Video Inpainting Under Constrained Camera Motion , 2007, IEEE Transactions on Image Processing.

[25]  Kiriakos N. Kutulakos,et al.  Confocal Stereo , 2006, International Journal of Computer Vision.

[26]  V. Bucha,et al.  AUGMENTED 3 D ENDOSCOPY VIDEO , 2008 .

[27]  Subhasis Chaudhuri,et al.  Depth From Defocus: A Real Aperture Imaging Approach , 1999, Springer New York.

[28]  V. Bucha,et al.  Augmented 3D Endoscopy Video , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[29]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[30]  Subhasis Chaudhuri,et al.  Depth From Defocus in Presence of Partial Self Occlusion , 2001, ICCV.

[31]  Alex Pentland,et al.  A New Sense for Depth of Field , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[33]  Frédo Durand,et al.  Defocus video matting , 2005, ACM Trans. Graph..

[34]  Stefano Soatto,et al.  Seeing beyond occlusions (and other marvels of a finite lens aperture) , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[35]  Ruigang Yang,et al.  Stereoscopic inpainting: Joint color and depth completion from stereo images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.