Occlusion-robust sensing method by using the light-field of a 3D display system toward interaction with a 3D image.

The recent developments in 3D display technology are remarkable. Owing to such developments, the importance of methods for interacting with 3D images has been increasing. In particular, the ability to perform input operations by directly touching displayed images is important. Therefore, a sensing method is required for recognizing the 3D positions of fingers and determining whether the fingers are touching the 3D image displayed. Conventionally, such a sensing method generally involves position sensing of fingers through image-based active sensing. However, this does not solve the following two problems: (1) the problem of positional registration, that is, the accurate matching between the displayed image and input location, and (2) the problem of occlusion robustness, that is, the achievement of successful sensing even with the presence of an occluding object between the sensor and hands to allow free movement of the hands. Our proposed method solves these problems through the following two ideas. First, we used a method called aerial imaging by retroreflection, which focuses light rays from a wider range than other 3D display systems. Second, for capturing the reflected light as well as projecting to the sensing target in active sensing, we used the light-field formed by the 3D display system. We also propose a method for obtaining rotation information, considering that the light-field formed by the reflected light changes based on the angle of the sensing target. Simulations and experiments were performed to evaluate the proposed system, and the system setup was optimized through simulations. In the first two experiments conducted on position and rotation sensing, we evaluated the sensing errors due to occlusion and found them to be less than 1.74 mm and less than 15.4 deg, respectively. In the third experiment, we constructed an interaction system with 3D images by using the proposed method and evaluated this system.

[1]  Myungjin Cho,et al.  Three-Dimensional Optical Sensing and Visualization Using Integral Imaging , 2011, Proceedings of the IEEE.

[2]  Shiro Suyama,et al.  Floating aerial LED signage based on aerial imaging by retro-reflection (AIRR). , 2014, Optics express.

[3]  Hamid Aghajan,et al.  Human detection in occluded scenes through optically inspired multi-camera image fusion. , 2017, Journal of the Optical Society of America. A, Optics, image science, and vision.

[4]  Masahiro Yamaguchi,et al.  3D touchable holographic light-field display. , 2016, Applied optics.

[5]  Paul M. Griffin,et al.  Generation of uniquely encoded light patterns for range data acquisition , 1992, Pattern Recognit..

[6]  Bahram Javidi,et al.  Photoelastic Analysis of Partially Occluded Objects With an Integral-Imaging Polariscope , 2014, Journal of Display Technology.

[7]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[8]  Dong-Hak Shin,et al.  Occlusion removal technique for improved recognition of partially occluded 3D objects in computational integral imaging , 2008 .

[9]  Gilbae Park,et al.  Reconstruction of three-dimensional occluded object using optical flow and triangular mesh reconstruction in integral imaging. , 2010, Optics express.

[10]  Eun-Soo Kim,et al.  Occlusion removal method of partially occluded object using variance in computational integral imaging , 2010 .

[11]  R. Raskar,et al.  Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging , 2012, Nature Communications.

[12]  Bahram Javidi,et al.  Three-Dimensional Imaging for Creating Real-World-Like Environments , 2013, Proceedings of the IEEE.

[13]  Byung-Gook Lee,et al.  Image quality enhancement of computational integral imaging reconstruction for partially occluded objects using binary weighting mask on occlusion areas. , 2011, Applied optics.

[14]  Chenfei Jin,et al.  Recovering three-dimensional shape through a small hole using three laser scatterings. , 2015, Optics letters.

[15]  Bahram Javidi,et al.  Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]. , 2013, Applied optics.