Foreground and shadow occlusion handling for outdoor augmented reality

Occlusion handling in augmented reality (AR) applications is challenging in synthesizing virtual objects correctly into the real scene with respect to existing foregrounds and shadows. Furthermore, outdoor environment makes the task more difficult due to the unpredictable illumination changes. This paper proposes novel outdoor illumination constraints for resolving the foreground occlusion problem in outdoor environment. The constraints can be also integrated into a probabilistic model of multiple cues for a better segmentation of the foreground. In addition, we introduce an effective method to resolve the shadow occlusion problem by using shadow detection and recasting with a spherical vision camera. We have applied the system in our digital cultural heritage project named Virtual Asuka (VA) and verified the effectiveness of the system.

[1]  Harry Shum,et al.  Background Cut , 2006, ECCV.

[2]  Liyuan Li,et al.  Robust change detection by fusing intensity and texture differences , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[3]  Cheng Lu,et al.  Shadow Segmentation and Shadow-Free Chromaticity via Markov Random Fields , 2005, Color Imaging Conference.

[4]  Pascal Fua,et al.  Making Background Subtraction Robust to Sudden Illumination Changes , 2008, ECCV.

[5]  D. B. Judd,et al.  Spectral Distribution of Typical Daylight as a Function of Correlated Color Temperature , 1964 .

[6]  Paul L. Rosin.  Thresholding for change detection , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[7]  Katsushi Ikeuchi,et al.  Consistent surface color for texturing large objects in outdoor scenes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[8]  Tetsuya Kakuta,et al.  Detection of moving objects and cast shadows using a spherical vision camera for outdoor mixed reality , 2008, VRST '08.

[9]  A. Criminisi,et al.  Bilayer Segmentation of Live Video , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[10]  David L. Olson,et al.  Advanced Data Mining Techniques , 2008 .

[11]  J. Marchant,et al.  Shadow-invariant classification for scenes illuminated by daylight. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[12]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Tetsuya Kakuta,et al.  Shading and shadowing of architecture in mixed reality , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[14]  Alexei A. Efros,et al.  Estimating natural illumination from a single outdoor image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[15]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[17]  Dar-Shyang Lee,et al.  Effective Gaussian mixture learning for video background subtraction , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Y. J. Tejwani,et al.  Robot vision , 1989, IEEE International Symposium on Circuits and Systems,.

[19]  Vincent Lepetit,et al.  Retexturing in the Presence of Complex Illumination and Occlusions , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[20]  Naokazu Yokoya,et al.  Geometric and photometric registration for real-time augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[21]  Berthold K. P. Horn,et al.  "Determining optical flow": A Retrospective , 1993, Artif. Intell..

[22]  Michael F. Cohen,et al.  Image and Video Matting: A Survey , 2007, Found. Trends Comput. Graph. Vis..

[23]  Robert O'Callaghan,et al.  Robust Change-Detection by Normalised Gradient-Correlation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Nassir Navab,et al.  Real-Time 3D Reconstruction for Occlusion-Aware Interactions in Mixed Reality , 2009, ISVC.

[25]  Kwanghoon Sohn,et al.  3D reconstruction of stereo images for interaction between real and virtual worlds , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..