High-speed 3D measurements at 20,000Hz with deep convolutional neural networks

Nowadays, the speed of ultra-fast photography can exceed one quadrillion. However, it can record only two-dimensional images which lack the depth information, greatly limiting our ability to perceive and to understand the complex real-world objects. Inspired by recent successes of deep learning methods in computer vision, we present a novel high-speed three-dimensional (3D) surface imaging approach named micro deep learning profilometry (μDLP) using the structured light illumination. With a properly trained deep neural network, the phase information is predicted from a single fringe image and then can be converted into the 3D shape. Our experiments demonstrate that μDLP can faithfully retrieve the geometry of dynamic objects at 20,000 frames per second. Moreover, comparative results show that μDLP has superior performance in terms of the phase accuracy, reconstruction efficiency, and the ease of implementation over widely used Fourier-transform-based fast 3D imaging techniques, verifying that μDLP is a powerful high-speed 3D surface imaging approach.

[1]  Jong-Hoon Nam,et al.  Localization of inner hair cell mechanotransducer channels using high-speed calcium imaging , 2009, Nature Neuroscience.

[2]  Anand Asundi,et al.  Review of phase measuring deflectometry , 2018, Optics and Lasers in Engineering.

[3]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[4]  T. Hewett,et al.  Reliability of landing 3D motion analysis: implications for longitudinal analyses. , 2007, Medicine and science in sports and exercise.

[5]  Qianbing Zhang,et al.  High-Speed Photography and Digital Optical Measurement Techniques for Geomaterials: Fundamentals and Applications , 2017, Rock Mechanics and Rock Engineering.

[6]  B. Jalali,et al.  Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena , 2009, Nature.

[7]  Lei Huang,et al.  Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review , 2016 .

[8]  Georg Weidenspointner,et al.  Femtosecond and nanometre visualization of structural dynamics in superheated nanoparticles , 2016, Nature Photonics.

[9]  Song Zhang,et al.  High-speed 3D shape measurement with structured light methods: A review , 2018, Optics and Lasers in Engineering.

[10]  Sam Van der Jeught,et al.  Real-time structured light profilometry: a review , 2016 .

[11]  Qian Chen,et al.  Robust dynamic 3-D measurements with motion-compensated phase-shifting profilometry , 2018 .

[12]  Ramesh Raskar,et al.  Slow art with a trillion frames per second camera , 2011, SIGGRAPH '11.

[13]  Zonghua Zhang,et al.  Generic exponential fringe model for alleviating phase error in phase measuring profilometry , 2018, Optics and Lasers in Engineering.

[14]  A. Tünnermann,et al.  High-speed three-dimensional shape measurement using GOBO projection , 2016 .

[15]  Xiang Peng,et al.  Light field 3D measurement using unfocused plenoptic cameras. , 2018, Optics letters.

[16]  Anand K. Asundi,et al.  Micro Fourier Transform Profilometry (μFTP): 3D shape measurement at 10, 000 frames per second , 2017, ArXiv.

[17]  Chiye Li,et al.  Single-shot compressed ultrafast photography at one hundred billion frames per second , 2014, Nature.

[18]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[19]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Zhiwei Xiong,et al.  Computational Depth Sensing : Toward high-performance commodity depth cameras , 2017, IEEE Signal Processing Magazine.

[21]  Xiubao Sui,et al.  Optimized pulse width modulation pattern strategy for three-dimensional profilometry with projector defocusing. , 2012, Applied optics.

[22]  Euripides G. M. Petrakis,et al.  A survey on industrial vision systems, applications, tools , 2003, Image Vis. Comput..

[23]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[24]  Anand Asundi,et al.  Two-dimensional digital image correlation for in-plane displacement and strain measurement: a review , 2009 .

[25]  Jianxiong Xiao,et al.  SUN RGB-D: A RGB-D scene understanding benchmark suite , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Yu Oishi,et al.  Sequentially timed all-optical mapping photography (STAMP) , 2014, Nature Photonics.

[27]  Liang Zhang,et al.  Fringe pattern analysis using deep learning , 2018, Advanced Photonics.

[28]  Justin Lee,et al.  Lensless computational imaging through deep learning , 2017, ArXiv.

[29]  Mumin Song,et al.  Overview of three-dimensional shape measurement using optical methods , 2000 .

[30]  Yibo Zhang,et al.  Deep Learning Microscopy , 2017, ArXiv.

[31]  Qian Chen,et al.  Phase shifting algorithms for fringe projection profilometry: A review , 2018, Optics and Lasers in Engineering.

[32]  Qian Chen,et al.  Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system. , 2016, Optics express.

[33]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[34]  Hongzhi Jiang,et al.  High dynamic range fringe acquisition: A novel 3-D scanning technique for high-reflective surfaces , 2012 .

[35]  Shijie Feng,et al.  General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique , 2014 .

[36]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[37]  Pedro Arias,et al.  Metrological comparison between Kinect I and Kinect II sensors , 2015 .

[38]  Zhang Liang,et al.  High dynamic range 3D measurements with fringe projection profilometry: a review , 2018 .