High-speed 3D shape measurement with structured light methods: A review

Abstract High-speed 3D shape measurement (or imaging) has seen tremendous growths over the past decades, especially the past few years due to the improved speed of computing devices and reduced costs of hardware components. 3D shape measurement technologies have started penetrating more into our daily lives than ever before with the recent release of iPhone X that has an built-in 3D sensor for Face ID, along with prior commercial success of inexpensive commercial sensors (e.g., Microsoft Kinect). This paper overviews the primary state-of-the-art 3D shape measurement techniques based on structured light methods, especially those that could achieve high measurement speed and accuracy. The fundamental principles behind those technologies will be elucidated, experimental results will be presented to demonstrate capabilities and/or limitations for those popular techniques, and finally present our perspectives on those remaining challenges to be conquered to make advanced 3D shape measurement techniques ubiquitous.

[1]  Dongliang Zheng,et al.  Ternary Gray code-based phase unwrapping for 3D measurement using binary patterns with projector defocusing. , 2017, Applied optics.

[2]  Beiwen Li,et al.  Superfast high-resolution absolute 3D recovery of a stabilized flapping flight process. , 2017, Optics express.

[3]  Peter Kühmstedt,et al.  Non-destructive 3D shape measurement of transparent and black objects with thermal fringes , 2016, Commercial + Scientific Sensing and Imaging.

[4]  Glen William Brooksby,et al.  Improving 3D surface measurement accuracy on metallic surfaces , 2005, SPIE Optical Metrology.

[5]  Tomáš Koutecký,et al.  Sensor planning system for fringe projection scanning of sheet metal parts , 2016 .

[6]  Jae-Sang Hyun,et al.  Pixel-wise absolute phase unwrapping using geometric constraints of structured light system. , 2016, Optics express.

[7]  Zhongwei Li,et al.  Fast phase measurement profilometry for arbitrary shape objects without phase unwrapping , 2013 .

[8]  Y. Wang,et al.  Absolute three-dimensional shape measurement using coded fringe patterns without phase unwrapping or projector calibration. , 2014, Optics express.

[9]  Xiubao Sui,et al.  Optimized pulse width modulation pattern strategy for three-dimensional profilometry with projector defocusing. , 2012, Applied optics.

[10]  Song Zhang,et al.  Superfast phase-shifting method for 3-D shape measurement. , 2010, Optics express.

[11]  Szymon Rusinkiewicz,et al.  Stripe boundary codes for real-time structured-light range scanning of moving objects , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[12]  Gastón A. Ayubi,et al.  Pulse-width modulation in defocused three-dimensional fringe projection. , 2010, Optics letters.

[13]  Xianyu Su,et al.  Automated phase-measuring profilometry using defocused projection of a Ronchi grating , 1992 .

[14]  Andrew Jones,et al.  Achieving eye contact in a one-to-many 3D video teleconferencing system , 2009, ACM Trans. Graph..

[15]  C. Waddington,et al.  Analysis of measurement sensitivity to illuminance and fringe-pattern gray levels for fringe-pattern projection adaptive to ambient lighting , 2010 .

[16]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[17]  Jake K. Aggarwal,et al.  Structure from stereo-a review , 1989, IEEE Trans. Syst. Man Cybern..

[18]  Feipeng Da,et al.  Adaptive fringe projection technique for high-dynamic range three-dimensional shape measurement using binary search , 2017 .

[19]  Xiangcheng Chen,et al.  Color-coding and phase-shift method for absolute phase measurement , 2013 .

[20]  Song Zhang,et al.  Natural method for three-dimensional range data compression. , 2013, Applied optics.

[21]  Song Zhang,et al.  Phase-optimized dithering technique for high-quality 3D shape measurement , 2013 .

[22]  Youfu Li,et al.  High-quality binary fringe generation via joint optimization on intensity and phase , 2017 .

[23]  Emanuele Zappa,et al.  Fourier-transform profilometry calibration based on an exhaustive geometric model of the system , 2009 .

[24]  Anand Asundi,et al.  Comparison of Fourier transform, windowed Fourier transform, and wavelet transform methods for phase extraction from a single fringe pattern in fringe projection profilometry , 2010 .

[25]  Robert A. Hummel,et al.  Experiments with the intensity ratio depth sensor , 1985, Comput. Vis. Graph. Image Process..

[26]  Fu-Pen Chiang,et al.  Trapezoidal phase-shifting method for three-dimensional shape measurement , 2005 .

[27]  Peisen S. Huang,et al.  3-D shape measurement by use of a modified Fourier transform method , 2008, Optical Engineering + Applications.

[28]  Tessa A. Hadlock,et al.  Novel 3-D video for quantification of facial movement , 2008, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[29]  Song Zhang,et al.  Multiwavelength depth encoding method for 3D range geometry compression. , 2015, Applied optics.

[30]  Peisen S. Huang,et al.  Novel method for structured light system calibration , 2006 .

[31]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[32]  Masayuki Yamamoto,et al.  Three-dimensional profilometry using liquid crystal grating , 2003, Other Conferences.

[33]  Wenjing Chen,et al.  Improved Fourier-transform profilometry. , 2007, Applied optics.

[34]  Alan C. Bovik,et al.  Modeling and quality assessment of halftoning by error diffusion , 2000, IEEE Trans. Image Process..

[35]  Qican Zhang,et al.  Virtual structured-light coding for three-dimensional shape data compression , 2012 .

[36]  Faramarz Farahi,et al.  Dynamics range enhancement in digital fringe projection technique , 2015 .

[37]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[38]  Qican Zhang,et al.  Universal calculation formula and calibration method in Fourier transform profilometry. , 2010, Applied optics.

[39]  C.-C. Jay Kuo,et al.  Technologies for 3D mesh compression: A survey , 2005, J. Vis. Commun. Image Represent..

[40]  Song Zhang,et al.  Optimal pulse width modulation for sinusoidal fringe generation with projector defocusing. , 2010, Optics letters.

[41]  Xianyu Su,et al.  Reliability-guided phase unwrapping algorithm: a review ☆ , 2004 .

[42]  Song Zhang,et al.  3D range geometry video compression with the H.264 codec , 2013 .

[43]  Song Zhang,et al.  Comparison of the squared binary, sinusoidal pulse width modulation, and optimal pulse width modulation methods for three-dimensional shape measurement with projector defocusing. , 2012, Applied optics.

[44]  Shijie Feng,et al.  High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection , 2013 .

[45]  Song Zhang,et al.  High-speed 3D imaging with digital fringe projection techniques , 2016, Optical Engineering + Applications.

[46]  Song Zhang,et al.  Composite phase-shifting algorithm for absolute phase measurement , 2012 .

[47]  Huijie Zhao,et al.  Rapid in-situ 3D measurement of shiny object based on fast and high dynamic range digital fringe projector , 2014 .

[48]  Yajun Wang,et al.  Three-dimensional shape measurement with binary dithered patterns. , 2012, Applied optics.

[49]  HANG,et al.  Superfast high-resolution absolute 3 D recovery of a stabilized flapping flight process , 2017 .

[50]  Jae-Sang Hyun,et al.  High-speed and high-accuracy 3D surface measurement using a mechanical projector. , 2018, Optics express.

[51]  Qican Zhang,et al.  Dynamic 3-D shape measurement method: A review , 2010 .

[52]  X. Su,et al.  Improved Fourier transform profilometry for the automatic measurement of 3D object shapes , 1990 .

[53]  Chenggen Quan,et al.  A modified phase-coding method for absolute phase retrieval , 2016 .

[54]  X. Su,et al.  High-speed optical measurement for the drumhead vibration. , 2005, Optics express.

[55]  Hongzhi Jiang,et al.  High dynamic range fringe acquisition: A novel 3-D scanning technique for high-reflective surfaces , 2012 .

[56]  Kevin George Harding,et al.  Shiny parts measurement using color separation , 2005, SPIE Optics East.

[57]  Qi Hao,et al.  Dual-frequency pattern scheme for high-speed 3-D shape measurement. , 2010, Optics express.

[58]  Andrew W. Fitzgibbon,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[59]  Song Zhang,et al.  Phase error compensation for three-dimensional shape measurement with projector defocusing. , 2011, Applied optics.

[60]  H. Rottenkolber,et al.  Two-Beam Interferometer Using a Laser , 1967 .

[61]  Song Zhang,et al.  Holovideo: Real-time 3D range video encoding and decoding on GPU , 2012 .

[62]  Zhaoyang Wang,et al.  Three-dimensional shape measurement with an arbitrarily arranged fringe projection profilometry system. , 2007, Optics letters.

[63]  J C Wyant,et al.  Two-wavelength phase shifting interferometry. , 1984, Applied optics.

[64]  Song Zhang,et al.  GPU-assisted high-resolution, real-time 3-D shape measurement. , 2006, Optics express.

[65]  Zhenkun Lei,et al.  An improved stair phase encoding method for absolute phase retrieval , 2015 .

[66]  William Cruz-Santos,et al.  Implicit absolute phase retrieval in digital fringe projection without reference lines , 2015 .

[67]  Hui Li,et al.  Enhanced high dynamic range 3D shape measurement based on generalized phase-shifting algorithm , 2016, ArXiv.

[68]  Song Zhang,et al.  High-resolution, real-time 3D absolute coordinate measurement based on a phase-shifting method. , 2006, Optics express.

[69]  HANG,et al.  High-speed and high-accuracy 3 D surface measurement using a mechanical projector , 2018 .

[70]  Jesús Villa,et al.  Transformation of phase to (x,y,z)-coordinates for the calibration of a fringe projection profilometer , 2012 .

[71]  Lei Huang,et al.  Least-squares calibration method for fringe projection profilometry considering camera lens distortion. , 2010, Applied optics.

[72]  Walter M. Duncan,et al.  Emerging digital micromirror device (DMD) applications , 2003, SPIE MOEMS-MEMS.

[73]  Anand Asundi,et al.  Fast three-dimensional measurements for dynamic scenes with shiny surfaces , 2017 .

[74]  Yajun Wang,et al.  3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique. , 2013, Optics express.

[75]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[76]  J. Jones,et al.  Optimum frequency selection in multifrequency interferometry. , 2003, Optics letters.

[77]  C. Waddington,et al.  Camera-independent saturation avoidance in measuring high-reflectivity-variation surfaces using pixel-wise composed images from projected patterns of different maximum gray level , 2014 .

[78]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[79]  Beiwen Li,et al.  Novel calibration method for structured-light system with an out-of-focus projector. , 2014, Applied optics.

[80]  Lei Huang,et al.  Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review , 2016 .

[81]  Ciyuan Qiu,et al.  Wavelength tracking with thermally controlled silicon resonators. , 2011, Optics express.

[82]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[83]  Qian Kemao,et al.  Windowed Fourier transform for fringe pattern analysis. , 2004, Applied optics.

[84]  Song Zhang,et al.  Flexible 3-D shape measurement using projector defocusing. , 2009, Optics letters.

[85]  Zonghua Zhang,et al.  Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques , 2012 .

[86]  Pierre Alliez,et al.  Recent advances in compression of 3D meshes , 2005, 2005 13th European Signal Processing Conference.

[87]  Shenghui Zhao,et al.  Accurate Dynamic 3D Sensing With Fourier-Assisted Phase Shifting , 2015, IEEE Journal of Selected Topics in Signal Processing.

[88]  Yajun Wang,et al.  Three-bit representation of three-dimensional range data. , 2013, Applied optics.

[89]  Jorge L Flores,et al.  Three-dimensional shape profiling by out-of-focus projection of colored pulse width modulation fringe patterns. , 2017, Applied optics.

[90]  Song Zhang,et al.  Superfast multifrequency phase-shifting technique with optimal pulse width modulation. , 2011, Optics express.

[91]  Russell W. Porter,et al.  Ronchi's Method of Optical Testing , 1929 .

[92]  Yajun Wang,et al.  Novel phase-coding method for absolute phase retrieval. , 2012, Optics letters.

[93]  Qingying Hu,et al.  Calibration of a three-dimensional shape measurement system , 2003 .

[94]  Song Zhang.  Three-dimensional range data compression using computer graphics rendering pipeline. , 2012, Applied optics.

[95]  Song Zhang,et al.  High-quality 3D shape measurement using saturated fringe patterns , 2016 .

[96]  Antonios Gasteratos,et al.  Review of Stereo Vision Algorithms: From Software to Hardware , 2008 .

[97]  Nikolaus Karpinsky,et al.  Composite phase-shifting algorithm for three-dimensional shape compression , 2010 .

[98]  Joaquim Salvi,et al.  A state of the art in structured light patterns for surface profilometry , 2010, Pattern Recognit..

[99]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[100]  LI Ze-ren,et al.  3-D profilometry for the impact process with marked fringes tracking , 2007 .

[101]  Zhaoyang Wang,et al.  Hyper-accurate flexible calibration technique for fringe-projection-based three-dimensional imaging , 2012 .

[102]  Yajun Wang,et al.  Two-channel high-accuracy Holoimage technique for three-dimensional data compression , 2016 .

[103]  Xukun Shen,et al.  PM-PM: PatchMatch With Potts Model for Object Segmentation and Stereo Matching , 2015, IEEE Transactions on Image Processing.

[104]  Jing Xu,et al.  High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics , 2017, International Journal of Intelligent Robotics and Applications.

[105]  V. Zayets,et al.  Magnetization-dependent loss in an (Al,Ga)As optical waveguide with an embedded Fe micromagnet. , 2010, Optics letters.

[106]  M. Takeda,et al.  Fourier transform profilometry for the automatic measurement of 3-D object shapes. , 1983, Applied optics.

[107]  Song Zhang,et al.  Flexible 3D shape measurement using projector defocusing: extended measurement range. , 2010, Optics letters.

[108]  Yunhui Yan,et al.  Fast 3D shape measurement using Fourier transform profilometry without phase unwrapping , 2016 .

[109]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[110]  Jonathan Kofman,et al.  Two-step triangular-pattern phase-shifting method for three-dimensional object-shape measurement , 2007 .

[111]  Song Zhang,et al.  High-resolution, real-time three-dimensional shape measurement , 2006 .

[112]  James C. Wyant,et al.  Surface Profilers, Multiple Wavelength, and White Light Intereferometry , 2006 .

[113]  Szymon Rusinkiewicz,et al.  Spacetime stereo: a unifying framework for depth from triangulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[114]  X.-X. Li,et al.  High-quality fringe pattern generation based on binary pattern optimization with projector defocusing , 2017 .

[115]  Irwin Sobel,et al.  On Calibrating Computer Controlled Cameras for Perceiving 3-D Scenes , 1973, IJCAI.

[116]  Yingchun Wu,et al.  Improved algorithm for phase-to-height mapping in phase measuring profilometry. , 2012, Applied optics.

[117]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[118]  Kai Liu,et al.  Accurate identification of saturated pixels for high dynamic range measurement , 2015 .

[119]  Qican Zhang,et al.  Dynamic 3-D shape measurement techniques with marked fringes tracking , 2009 .

[120]  Céline Hudelot,et al.  3D Mesh Compression , 2015, ACM Comput. Surv..

[121]  Hualie Jiang,et al.  A high dynamic range structured light means for the 3D measurement of specular surface , 2017 .

[122]  Qian Chen,et al.  Improved intensity-optimized dithering technique for 3D shape measurement , 2015 .

[123]  Gastón A. Ayubi,et al.  Three-dimensional profiling with binary fringes using phase-shifting interferometry algorithms. , 2011, Applied optics.

[124]  Peter Kühmstedt,et al.  Array projection of aperiodic sinusoidal fringes for high-speed three-dimensional shape measurement , 2014 .

[125]  Christian Bräuer-Burchardt,et al.  Code Minimization for Fringe Projection Based 3D Stereo Sensors by Calibration Improvement , 2014, ArXiv.

[126]  Jing Xu,et al.  Method for large-range structured light system calibration. , 2016, Applied optics.

[127]  Peisen S. Huang,et al.  Absolute phase technique for the Fourier transform method , 2009 .

[128]  J. Wyant,et al.  Multiple-wavelength phase-shifting interferometry. , 1985, Applied optics.

[129]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[130]  Xianyu Su,et al.  Accurate and fast 3D surface measurement with temporal-spatial binary encoding structured illumination. , 2016, Optics express.

[131]  Song Zhang,et al.  Superfast 3D absolute shape measurement using five binary patterns , 2017 .

[132]  Fu-Pen Chiang,et al.  Color phase-shifting technique for three-dimensional shape measurement , 2006 .

[133]  Song Zhang,et al.  Genetic method to optimize binary dithering technique for high-quality fringe generation. , 2013, Optics letters.

[134]  Dongliang Zheng,et al.  Phase coding method for absolute phase retrieval with a large number of codewords. , 2012, Optics express.

[135]  X. Su,et al.  Method for improving sinusoidal quality of error diffusion binary encoded fringe used in phase measurement profilometry , 2016 .

[136]  Hui Lin,et al.  Adaptive digital fringe projection technique for high dynamic range three-dimensional shape measurement. , 2016, Optics express.

[137]  Xiaohui Zhou,et al.  Multiview phase shifting: a full-resolution and high-speed 3D measurement framework for arbitrary shape dynamic objects. , 2013, Optics letters.

[138]  Song Zhang.  Recent progresses on real-time 3D shape measurement using digital fringe projection techniques , 2010 .

[139]  Shijie Feng,et al.  General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique , 2014 .

[140]  G Sansoni,et al.  Three-dimensional vision based on a combination of gray-code and phase-shift light projection: analysis and compensation of the systematic errors. , 1999, Applied optics.

[141]  Song Zhang,et al.  Autoexposure for three-dimensional shape measurement using a digital-light-processing projector , 2011 .

[142]  Song Zhang,et al.  Digital sinusoidal fringe pattern generation: Defocusing binary patterns VS focusing sinusoidal patterns , 2010 .

[143]  B Budianto,et al.  Marker encoded fringe projection profilometry for efficient 3D model acquisition. , 2014, Applied optics.

[144]  S. H. Rowe,et al.  Surface Topography of Non-optical Surfaces by Projected Interference Fringes , 1967, Nature.

[145]  A. Tünnermann,et al.  High-speed three-dimensional shape measurement using GOBO projection , 2016 .

[146]  Xiaosheng Cheng,et al.  A flexible phase-shifting method with absolute phase marker retrieval , 2012 .

[147]  Toru Yoshizawa,et al.  Liquid crystal grating for profilometry using structured light , 2005, SPIE Optics East.

[148]  Beiwen Li,et al.  Intensity-optimized dithering technique for three-dimensional shape measurement with projector defocusing , 2014 .

[149]  Song Zhang,et al.  3D shape measurement with 2D area modulated binary patterns , 2012 .

[150]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[151]  Shiping Zhu,et al.  Local stereo matching algorithm with efficient matching cost and adaptive guided image filter , 2017, The Visual Computer.

[152]  Sam Van der Jeught,et al.  Real-time structured light profilometry: a review , 2016 .

[153]  Jing Xu,et al.  Method for out-of-focus camera calibration. , 2016, Applied optics.

[154]  Duane C. Brown,et al.  Close-Range Camera Calibration , 1971 .

[155]  Mitsuo Takeda,et al.  Fourier fringe analysis and its application to metrology of extreme physical phenomena: a review [Invited]. , 2013, Applied optics.

[156]  Hui Lin,et al.  Three-dimensional shape measurement technique for shiny surfaces by adaptive pixel-wise projection intensity adjustment , 2017 .

[157]  X Su,et al.  Area modulation grating for sinusoidal structure illumination on phase-measuring profilometry. , 2001, Applied optics.

[158]  Song Zhang,et al.  High dynamic range real-time 3D shape measurement. , 2016, Optics express.

[159]  Michael Frankfurter,et al.  Optical Shop Testing , 2016 .

[160]  Radim Sára,et al.  Stratified Dense Matching for Stereopsis in Complex Scenes , 2003, BMVC.

[161]  Song Zhang,et al.  Double-pattern triangular pulse width modulation technique for high-accuracy high-speed 3D shape measurement. , 2017, Optics express.

[162]  Luc Van Gool,et al.  Fast 3D Scanning with Automatic Motion Compensation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[163]  Beiwen Li,et al.  Microscopic structured light 3D profilometry: Binary defocusing technique vs. sinusoidal fringe projection , 2017 .

[164]  Qian Kemao.  Applications of windowed Fourier fringe analysis in optical measurement: A review , 2015 .

[165]  Hongwei Guo,et al.  Least-squares fitting of carrier phase distribution by using a rational function in fringe projection profilometry , 2006 .

[166]  Song Zhang,et al.  High-quality fringe pattern generation using binary pattern optimization through symmetry and periodicity , 2014 .

[167]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[168]  Basel Salahieh,et al.  Multi-polarization fringe projection imaging for high dynamic range objects. , 2014, Optics express.

[169]  Dennis C. Ghiglia,et al.  Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software , 1998 .

[170]  Qian Kemao,et al.  Two-dimensional windowed Fourier transform for fringe pattern analysis: Principles, applications and implementations , 2007 .