3D Surface reconstruction of transparent objects using laser scanning with LTFtF method

Abstract Three-dimensional (3D) vision plays an important role in robotics and industrial automation, wherein the reconstruction of transparent objects poses a great challenge. In this paper, we present a scheme for reconstructing the 3D surfaces of transparent objects based on a combination of stereo matching and laser scanning. The traditional laser scanning method can achieve good results in the 3D reconstruction of opaque objects. However, this method will fail for transparent and semitransparent objects because both the front and back surfaces of these objects will reflect the feature laser line. It is difficult to determine the reflection surface of the feature lines captured by the camera. To distinguish the laser reflected by the front surface from the distracting laser reflections by the other surfaces, a laser tracking frame to frame (LTFtF) method is proposed. The laser lines reflected from the front surface of the target are then reconstructed using stereo vision. The entire 3D surface is obtained through laser scanning. In addition, we improve the matching accuracy and point cloud density through an internal pixel matching method. Various experiments were conducted to verify the capability of the proposed method, which includes the reconstruction of transparent and semitransparent objects of various shapes. The proposed method was compared with traditional laser scanning algorithms.

[1]  Sebastian Bodenstedt,et al.  Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. , 2012, Medical physics.

[2]  Antonios Gasteratos,et al.  Review of Stereo Vision Algorithms: From Software to Hardware , 2008 .

[3]  Minglun Gong,et al.  Stereo-Based 3D Reconstruction of Dynamic Fluid Surfaces by Global Optimization , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Lei Huang,et al.  Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review , 2016 .

[5]  Jing Xu,et al.  Status, challenges, and future perspectives of fringe projection profilometry , 2020 .

[6]  Ken Chen,et al.  Normal Direction Measurement and Optimization With a Dense Three-Dimensional Point Cloud in Robotic Drilling , 2018, IEEE/ASME Transactions on Mechatronics.

[7]  Carsten Steger,et al.  An Unbiased Detector of Curvilinear Structures , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  A. Tünnermann,et al.  High-speed three-dimensional shape measurement using GOBO projection , 2016 .

[9]  Qian Chen,et al.  Phase shifting algorithms for fringe projection profilometry: A review , 2018, Optics and Lasers in Engineering.

[10]  Xinyang Xu,et al.  Reconstructing the surface of transparent objects by polarized light measurements. , 2017, Optics express.

[11]  Jingyi Yu,et al.  Reconstructing Gas Flows Using Light-Path Approximation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Gregory Hager,et al.  Vision-based navigation in image-guided interventions. , 2011, Annual review of biomedical engineering.

[13]  Feng Li,et al.  Dynamic fluid surface acquisition using a camera array , 2011, 2011 International Conference on Computer Vision.

[14]  Shih-Fu Chang,et al.  Model-Driven Feedforward Prediction for Manipulation of Deformable Objects , 2016, IEEE Transactions on Automation Science and Engineering.

[15]  Hans-Peter Seidel,et al.  DISCO: acquisition of translucent objects , 2004, ACM Trans. Graph..

[16]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[17]  Michael Figl,et al.  Reconstruction of a 3D surface from video that is robust to missing data and outliers: Application to minimally invasive surgery using stereo and mono endoscopes , 2012, Medical Image Anal..

[18]  Song Zhang.  Recent progresses on real-time 3D shape measurement using digital fringe projection techniques , 2010 .

[19]  Kiriakos N. Kutulakos,et al.  Reconstructing the Surface of Inhomogeneous Transparent Scenes by Scatter-Trace Photography , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[20]  Ding Liu,et al.  Frequency-Based 3D Reconstruction of Transparent and Specular Objects , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Minglun Gong,et al.  3D Reconstruction of Transparent Objects with Position-Normal Consistency , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Kai Han,et al.  A fixed viewpoint approach for dense reconstruction of transparent objects , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jordi Llorens,et al.  Kinect v2 Sensor-Based Mobile Terrestrial Laser Scanner for Agricultural Outdoor Applications , 2017, IEEE/ASME Transactions on Mechatronics.

[24]  Gabriel Taubin,et al.  Simple, Accurate, and Robust Projector-Camera Calibration , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[25]  Dani Lischinski,et al.  Differentiable refraction-tracing for mesh reconstruction of transparent objects , 2020, ACM Trans. Graph..

[26]  Franc¸ois Blais,et al.  Review of 20 years of range sensor development , 2003, IS&T/SPIE Electronic Imaging.

[27]  Congying Sui,et al.  3D reconstruction of objects with occlusion and surface reflection using a dual monocular structured light system. , 2020, Applied optics.

[28]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  F. Mériaudeau,et al.  Scanning from heating: 3D shape estimation of transparent objects from local surface heating. , 2009, Optics express.

[30]  Minglun Gong,et al.  Simultaneous 3D Reconstruction for Water Surface and Underwater Scene , 2018, ECCV.

[31]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[32]  Peisen S. Huang,et al.  Fast three-step phase-shifting algorithm , 2006 .

[33]  Congying Sui,et al.  Active Stereo 3-D Surface Reconstruction Using Multistep Matching , 2020, IEEE Transactions on Automation Science and Engineering.

[34]  Song Zhang,et al.  High-speed 3D shape measurement with structured light methods: A review , 2018, Optics and Lasers in Engineering.

[35]  Takayuki Okatani,et al.  Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center , 1997, Comput. Vis. Image Underst..

[36]  Minglun Gong,et al.  Full 3D reconstruction of transparent objects , 2018, ACM Trans. Graph..

[37]  Peter Kühmstedt,et al.  Non-destructive 3D shape measurement of transparent and black objects with thermal fringes , 2016, Commercial + Scientific Sensing and Imaging.

[38]  Mumin Song,et al.  Overview of three-dimensional shape measurement using optical methods , 2000 .