Dynamic 3D reconstruction improvement via intensity video guided 4D fusion

Abstract The availability of high-speed 3D video sensors has greatly facilitated 3D shape acquisition of dynamic and deformable objects, but high frame rate 3D reconstruction is always degraded by spatial noise and temporal fluctuations. This paper presents a simple yet powerful dynamic 3D reconstruction improvement algorithm based on intensity video guided multi-frame 4D fusion. Temporal tracking of intensity image points (of moving and deforming objects) allows registration of the corresponding 3D model points, whose 3D noise and fluctuations are then reduced by spatio-temporal multi-frame 4D fusion. We conducted simulated noise tests and real experiments on four 3D objects using a 1000 fps 3D video sensor. The results demonstrate that the proposed algorithm is effective at reducing 3D noise and is robust against intensity noise. It outperforms existing algorithms with good scalability on both stationary and dynamic objects.

[1]  Hui Lin,et al.  Depth image enhancement for Kinect using region growing and bilateral filter , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[2]  Dan Guo,et al.  Improved marching tetrahedra algorithm based on hierarchical signed distance field and multi-scale depth map fusion for 3D reconstruction , 2017, J. Vis. Commun. Image Represent..

[3]  Jing Wang,et al.  STV-based video feature processing for action recognition , 2013, Signal Process..

[4]  Partha Pratim Das,et al.  Characterizations of Noise in Kinect Depth Images: A Review , 2014, IEEE Sensors Journal.

[5]  Luis Salgado,et al.  Depth-Color Fusion Strategy for 3-D Scene Modeling With Kinect , 2013, IEEE Transactions on Cybernetics.

[6]  Masatoshi Ishikawa,et al.  High-speed 3D sensing with three-view geometry using a segmented pattern , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[7]  Jakob Wasza,et al.  Real-time preprocessing for dense 3-D range imaging on the GPU: Defect interpolation, bilateral temporal averaging and guided filtering , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[8]  Dmitriy Vatolin,et al.  Temporal filtering for depth maps generated by Kinect depth camera , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[9]  Roque Alfredo Osornio-Rios,et al.  A new method for inpainting of depth maps from time-of-flight sensors based on a modified closing by reconstruction algorithm , 2017, J. Vis. Commun. Image Represent..

[10]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[11]  Ernesto Damiani,et al.  Temporal Denoising of Kinect Depth Data , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[12]  Moonhong Baeg,et al.  Spatial Uncertainty Model for Visual Features Using a Kinect™ Sensor , 2012, Sensors.

[13]  J. Paul Siebert,et al.  Active Animate Stereo Vision , 1993, BMVC.

[14]  Wenjun Zeng,et al.  Kinect-like depth denoising , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[15]  Shaun J. Canavan,et al.  BP4D-Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database , 2014, Image Vis. Comput..

[16]  Lin Yang,et al.  Evaluating and Improving the Depth Accuracy of Kinect for Windows v2 , 2015, IEEE Sensors Journal.

[17]  Jian Sun,et al.  Guided Image Filtering , 2010, ECCV.

[18]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jian Xiang,et al.  Motion recognition and synthesis based on 3D sparse representation , 2015, Signal Process..

[20]  Robert B. Fisher,et al.  Performance characterization of a high-speed stereo vision sensor for acquisition of time-varying 3D shapes , 2010, Machine Vision and Applications.

[21]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[22]  Shahram Izadi,et al.  Modeling Kinect Sensor Noise for Improved 3D Reconstruction and Tracking , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[23]  Jan Kautz,et al.  PMBP: PatchMatch Belief Propagation for Correspondence Field Estimation , 2014, International Journal of Computer Vision.

[24]  Andrew Zisserman,et al.  MLESAC: A New Robust Estimator with Application to Estimating Image Geometry , 2000, Comput. Vis. Image Underst..

[25]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[26]  Andrew W. Fitzgibbon,et al.  3D scanning deformable objects with a single RGBD sensor , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Yonghong Song,et al.  A Shadow Repair Approach for Kinect Depth Maps , 2012, ACCV.