Fast spatial-temporal stereo matching for 3D face reconstruction under speckle pattern projection

Abstract Three-dimensional (3D) face reconstruction can be tackled in either measurement-based means or model-based means. The former requires special hardwares or devices, such as structured light setups. This paper addresses 3D face reconstruction by measurement-based means, more specifically a special kind of structured light called space–time speckle projection. Under such a setup, we propose a novel and efficient spatial–temporal stereo scheme towards fast and accurate 3D face recovery. To improve the overall computational efficiency, our scheme consists of a series of optimization strategies including face-cropping-based stereo matching, coarse-to-fine stereo matching strategy applied to face areas, and spatial–temporal integral image (STII) for accelerating the matching cost computation. Based on the results, the proposed scheme is able to reconstruct a 3D face in hundreds of milliseconds on a normal PC, and its performance is validated both qualitatively and quantitatively.

[1]  Lifeng Sun,et al.  Cross-Scale Cost Aggregation for Stereo Matching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Franklin C. Crow,et al.  Summed-area tables for texture mapping , 1984, SIGGRAPH.

[3]  Qican Zhang,et al.  High-speed 3D face measurement based on color speckle projection , 2015 .

[4]  Shai Avidan,et al.  Coherency Sensitive Hashing , 2011, ICCV.

[5]  Chenbo Shi,et al.  Depth estimation for speckle projection system using progressive reliable points growing matching. , 2013, Applied optics.

[6]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[7]  Qijun Zhao,et al.  Examplar coherent 3D face reconstruction from forensic mugshot database , 2017, Image Vis. Comput..

[8]  Weiming Dong,et al.  Segment-tree based cost aggregation for stereo matching with enhanced segmentation advantage , 2013, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Martin Schaffer,et al.  Statistical patterns: an approach for high-speed and high-accuracy shape measurements , 2014 .

[10]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Szymon Rusinkiewicz,et al.  Spacetime stereo: a unifying framework for depth from triangulation , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Changhe Zhou,et al.  3D human face reconstruction based on band-limited binary patterns , 2016 .

[13]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Thomas Vetter,et al.  Face Recognition Based on Fitting a 3D Morphable Model , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Yung-Sheng Chen,et al.  Measuring of a three-dimensional surface by use of a spatial distance computation. , 2003, Applied optics.

[16]  Junfei Dai,et al.  Absolute three-dimensional shape measurement with a known object. , 2017, Optics express.

[17]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[18]  Frank Dellaert,et al.  Structure from motion without correspondence , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[19]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[20]  Andreas Tünnermann,et al.  BRDF-dependent accuracy of array-projection-based 3D sensors. , 2017, Applied optics.

[21]  Adam Finkelstein,et al.  The Generalized PatchMatch Correspondence Algorithm , 2010, ECCV.

[22]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Feng Liu,et al.  Joint Face Alignment and 3D Face Reconstruction , 2016, ECCV.

[25]  Stefano Soatto,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE IEEE TRANSACTION OF PATTERN RECO , 2022 .

[26]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[27]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[28]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[29]  Andrew W. Fitzgibbon,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[30]  Martin Schaffer,et al.  3D shape measurement of static and moving objects with adaptive spatiotemporal correlation. , 2014, Applied optics.

[31]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[32]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[33]  Martin Schaffer,et al.  High-speed pattern projection for three-dimensional shape measurement using laser speckles. , 2010, Applied optics.

[34]  Enhua Wu,et al.  Constant Time Weighted Median Filtering for Stereo Matching and Beyond , 2013, 2013 IEEE International Conference on Computer Vision.

[35]  Heiko Hirschmüller,et al.  Evaluation of Stereo Matching Costs on Images with Radiometric Differences , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Pietro Parodi,et al.  3D Shape Reconstruction by Using Vanishing Points , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Charles T. Loop,et al.  Computing rectifying homographies for stereo vision , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[38]  Axel Wiegmann,et al.  Human face measurement by projecting bandlimited random patterns. , 2006, Optics express.

[39]  Atsuto Maki,et al.  Towards a simulation driven stereo vision system , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[40]  Joaquim Salvi,et al.  A state of the art in structured light patterns for surface profilometry , 2010, Pattern Recognit..

[41]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[42]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[43]  Antonio Criminisi,et al.  Shape from Texture: Homogeneity Revisited , 2000, BMVC.

[44]  Changhe Zhou,et al.  3D shape measurement of a ground surface optical element using band-pass random patterns projection , 2015 .

[45]  Carsten Rother,et al.  Fast Cost-Volume Filtering for Visual Correspondence and Beyond , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  A. Verri,et al.  A compact algorithm for rectification of stereo pairs , 2000 .

[47]  Stefano Soatto,et al.  A geometric approach to shape from defocus , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.