Detection of moving objects and cast shadows using a spherical vision camera for outdoor mixed reality

This paper presents a method to detect moving objects and remove their shadows for superimposing them on Mixed Reality (MR) systems. We cut out the foreground from a real image using a probability-based segmentation method. Using color, spatial, and temporal priors, we can improve the accuracy of the segmentation. Energy minimization is executed by graph cuts. Then we remove the shadow region from the foreground with F-value calculated from the pixel value and the spectral sensitivity characteristic of the camera. Finally we superimpose virtual objects using the stencil buffer, which is used to limit the area of rendering for each pixel. Synthesized images of an outdoor scene show the efficiency of the proposed method.

[1]  J. Marchant,et al.  Shadow-invariant classification for scenes illuminated by daylight. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[2]  K. Ikeuchi,et al.  Real-time Soft Shadows in Mixed Reality Using Shadowing Planes , 2007, MVA.

[3]  David S. Ebert,et al.  Visualization and computer graphics , 2007 .

[4]  Katsushi Ikeuchi,et al.  Acquiring a Radiance Distribution to Superimpose Virtual Objects onto Real Scene , 2001, MVA.

[5]  A. Criminisi,et al.  Bilayer Segmentation of Live Video , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  Naokazu Yokoya,et al.  A Stereo Vision-based Mixed Reality System with Natural Feature Point Tracking , 2004 .

[7]  Dirk Bartz,et al.  Occlusion handling for medical augmented reality using a volumetric phantom model , 2004, VRST '04.

[8]  Mark Billinghurst,et al.  The use of dense stereo range data in augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[9]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  G D Finlayson,et al.  Color constancy at a pixel. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[11]  Patrick Pérez,et al.  Interactive Image Segmentation Using an Adaptive GMMRF Model , 2004, ECCV.

[12]  Nicolas Martel-Brisson,et al.  Moving cast shadow detection from a Gaussian mixture shadow model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Gerald Schaefer,et al.  Solving for Colour Constancy using a Constrained Dichromatic Reflection Model , 2001, International Journal of Computer Vision.

[14]  Poorvi L. Vora,et al.  Digital color cameras - 2 - Spectral response , 1997 .

[15]  Harry Shum,et al.  Background Cut , 2006, ECCV.

[16]  Kwanghoon Sohn,et al.  3D reconstruction of stereo images for interaction between real and virtual worlds , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[17]  Vincent Lepetit,et al.  A semi-automatic method for resolving occlusion in augmented reality , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[18]  D. B. Judd,et al.  Spectral Distribution of Typical Daylight as a Function of Correlated Color Temperature , 1964 .

[19]  Katsushi Ikeuchi,et al.  Consistent surface color for texturing large objects in outdoor scenes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[20]  Andrew Blake,et al.  Bi-layer segmentation of binocular stereo video , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).