Handling occlusions in augmented reality based on 3D reconstruction method

The correct relationships between real and virtual objects are of utmost importance to a realistic augmented reality system, in which the occlusion handling method should be able to estimate the spatial relationships between real and virtual objects, as well as handle the mutual occlusion automatically in real-time. To accomplish the above tasks simultaneously, we propose a novel occlusion handling method based on 3D reconstruction, which consists of offline stage and online stage. In the offline stage, we get the depth map of the real scene using a low cost RGB-D camera. Then the 3D coordinate of each point in the global coordinate system are obtained and will be used in the online occlusion handling stage. In the online stage, we design a GPU based 3D point clouds alignment method by using point to tangent plane distance as error metric to accelerate the convergence speed and reduce the iterations. The correct relationships between real and virtual objects are then obtained automatically by comparing each pixel's Z coordinate value of real objects with that of virtual objects in a smaller region to achieve real-time performance. More specifically, we can judge and handle the mutual occlusion without human interactivity in real time, and experimental results prove its effectiveness.

[1]  Tiow Seng Tan,et al.  Resolving occlusion in image sequence made easy , 1998, The Visual Computer.

[2]  Junqing Yu,et al.  Efficient BOF Generation and Compression for On-Device Mobile Visual Location Recognition , 2014, IEEE MultiMedia.

[3]  Qi Tian,et al.  Mining flickr landmarks by modeling reconstruction sparsity , 2011, TOMCCAP.

[4]  Wen Gao,et al.  Learning to Distribute Vocabulary Indexing for Scalable Visual Search , 2013, IEEE Transactions on Multimedia.

[5]  Junqing Yu,et al.  Real-Time Camera Pose Estimation for Wide-Area Augmented Reality Applications , 2011, IEEE Computer Graphics and Applications.

[6]  Junqing Yu,et al.  On-Device Mobile Visual Location Recognition by Using Panoramic Images and Compressed Sensing Based Visual Descriptors , 2014, PloS one.

[7]  Soon-Yong Park,et al.  An accurate and fast point-to-plane registration technique , 2003, Pattern Recognit. Lett..

[8]  Takeshi Oishi,et al.  Reduction of contradictory partial occlusion in mixed reality by using characteristics of transparency perception , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[9]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[10]  Junqing Yu,et al.  On-Device Mobile Visual Location Recognition by Integrating Vision and Inertial Sensors , 2013, IEEE Transactions on Multimedia.

[11]  Qi Tian,et al.  Task-Dependent Visual-Codebook Compression , 2012, IEEE Transactions on Image Processing.

[12]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[13]  Jiejie Zhu,et al.  Handling occlusions in video‐based augmented reality using depth information , 2010, Comput. Animat. Virtual Worlds.

[14]  Cheng Wang,et al.  An automatic occlusion handling method in augmented reality , 2010 .

[15]  Dirk Bartz,et al.  Occlusion handling for medical augmented reality using a volumetric phantom model , 2004, VRST '04.

[16]  Vincent Lepetit,et al.  A semi-automatic method for resolving occlusion in augmented reality , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Cheng Wang,et al.  Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach , 2010, Sensors.

[18]  Vineet R. Kamat,et al.  Real-Time Occlusion Handling for Dynamic Augmented Reality Using Geometric Sensing and Graphical Shading , 2013 .

[19]  Vincent Lepetit,et al.  Handling occlusion in augmented reality systems: a semi-automatic method , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[20]  Suyang Dong,et al.  Visual Occlusion in Outdoor Augmented Reality Using TOF Camera and OpenGL Frame Buffer , 2010 .

[21]  Shana Smith,et al.  GPU-Based Real-Time Occlusion in an Immersive Augmented Reality Environment , 2009, J. Comput. Inf. Sci. Eng..

[22]  Michael Gervautz,et al.  Occlusion in collaborative augmented environments , 1999, Comput. Graph..

[23]  Soh-Khim Ong,et al.  Registration using natural features for augmented reality systems , 2006, IEEE Transactions on Visualization and Computer Graphics.

[24]  Vineet R. Kamat,et al.  Scalable Algorithm for Resolving Incorrect Occlusion in Dynamic Augmented Reality Engineering Environments , 2010, Comput. Aided Civ. Infrastructure Eng..

[25]  Shogo Nishida,et al.  Occlusion detection of real objects using contour based stereo matching , 2005, ICAT '05.

[26]  Junqing Yu,et al.  Projected Residual Vector Quantization for ANN Search , 2014, IEEE MultiMedia.

[27]  Wenzhi Chen,et al.  Handling occlusions in video-based augmented reality using depth information , 2010 .

[28]  Heinrich Niemann,et al.  Dense disparity maps in real-time with an application to augmented reality , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[29]  David E. Breen,et al.  Interactive Occlusion and Automatic Object Placement for Augmented Reality , 1996, Comput. Graph. Forum.

[30]  Romero Tori,et al.  Mutual occlusion between real and virtual elements in Augmented Reality based on fiducial markers , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[31]  Hideki Hashimoto,et al.  A representation of occlusion between real objects and virtual information in Intelligent Room - for AR , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.