Stereo image de-fencing using smartphones

Conventional approaches to image de-fencing have limited themselves to using only image data in adjacent frames of the captured video of an approximately static scene. In this work, we present a method to harness disparity using a stereo pair of fenced images in order to detect fence pixels. Tourists and amateur photographers commonly carry smartphones/phablets which can be used to capture a short video sequence of the fenced scene. We model the formation of the occluded frames in the captured video. Furthermore, we propose an optimization framework to estimate the de-fenced image using the total variation prior to regularize the ill-posed problem.

[1]  Ruigang Yang,et al.  Stereoscopic inpainting: Joint color and depth completion from stereo images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Yanxi Liu,et al.  Image de-fencing , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  William T. Freeman,et al.  A computational approach for obstruction-free photography , 2015, ACM Trans. Graph..

[4]  Yanxi Liu,et al.  A computational model for periodic pattern perception based on frieze and wallpaper groups , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[6]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Rajiv R. Sahay,et al.  My camera can see through fences: A deep learning approach for image de-fencing , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[8]  Christine Guillemot,et al.  Image Inpainting : Overview and Recent Advances , 2014, IEEE Signal Processing Magazine.

[9]  Wei Liu,et al.  Video De-Fencing , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Mohan S. Kankanhalli,et al.  Seeing through the fence: Image de-fencing using a video sequence , 2013, 2013 IEEE International Conference on Image Processing.

[11]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[12]  Sankaraganesh Jonna,et al.  Detection and removal of fence occlusions in an image using a video of the static/dynamic scene. , 2016, Journal of the Optical Society of America. A, Optics, image science, and vision.

[13]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[14]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Yuanjie Zheng,et al.  Learning based digital matting , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[18]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[19]  Yanxi Liu,et al.  Deformed Lattice Detection in Real-World Images Using Mean-Shift Belief Propagation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[21]  Mohan S. Kankanhalli,et al.  A multimodal approach for image de-fencing and depth inpainting , 2015, 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR).

[22]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[23]  Cheng Soon Ong,et al.  Multivariate spearman's ρ for aggregating ranks using copulas , 2016 .

[24]  Ping Tan,et al.  Automatic Fence Segmentation in Videos of Dynamic Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Sayali Baheti,et al.  Region Filling and Object Removal by Exemplar based Image Inpainting , 2015 .