DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time

We present the first dense SLAM system capable of reconstructing non-rigidly deforming scenes in real-time, by fusing together RGBD scans captured from commodity sensors. Our DynamicFusion approach reconstructs scene geometry whilst simultaneously estimating a dense volumetric 6D motion field that warps the estimated geometry into a live frame. Like KinectFusion, our system produces increasingly denoised, detailed, and complete reconstructions as more measurements are fused, and displays the updated model in real time. Because we do not require a template or other prior scene model, the approach is applicable to a wide range of moving objects and scenes.

[1]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[2]  B. Curless.  New Methods for Surface Reconstruction from Range Images , 1997 .

[3]  Szymon Rusinkiewicz,et al.  Non-rigid range-scan alignment using thin-plate splines , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[4]  Leonidas J. Guibas,et al.  Dynamic geometry registration , 2007, Symposium on Geometry Processing.

[5]  Jirí Zára,et al.  Skinning with dual quaternions , 2007, SI3D.

[6]  Szymon Rusinkiewicz,et al.  Global non-rigid alignment of 3-D scans , 2007, SIGGRAPH 2007.

[7]  Leonidas J. Guibas,et al.  Eurographics Symposium on Geometry Processing (2007) Reconstruction of Deforming Geometry from Time-varying Point Clouds , 2022 .

[8]  M. Pauly,et al.  Embedded deformation for shape manipulation , 2007, SIGGRAPH 2007.

[9]  Günther Greiner,et al.  Reconstructing Animated Meshes from Time‐Varying Point Clouds , 2008, Comput. Graph. Forum.

[10]  Dan A. Alcantara,et al.  Space-time surface reconstruction using incompressible flow , 2008, SIGGRAPH 2008.

[11]  Hao Li,et al.  Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans , 2008, Comput. Graph. Forum.

[12]  Hans-Peter Seidel,et al.  Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data , 2009, TOGS.

[13]  Matthias Zwicker,et al.  Range Scan Registration Using Reduced Deformable Models , 2009, Comput. Graph. Forum.

[14]  Leonidas J. Guibas,et al.  Robust single-view geometry and motion reconstruction , 2009, ACM Trans. Graph..

[15]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[16]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[17]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[18]  Pieter Peers,et al.  Temporally coherent completion of dynamic shapes , 2012, TOGS.

[19]  Hans-Peter Seidel,et al.  Animation cartography—intrinsic reconstruction of shape and motion , 2012, TOGS.

[20]  Marsette Vona,et al.  Moving Volume KinectFusion , 2012, BMVC.

[21]  John J. Leonard,et al.  Kintinuous: Spatially Extended KinectFusion , 2012, AAAI 2012.

[22]  Andrew W. Fitzgibbon,et al.  The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Xinguo Liu,et al.  A memory-efficient kinectfusion using octree , 2012, CVM.

[24]  Jan-Michael Frahm,et al.  Scanning and tracking dynamic objects with commodity depth cameras , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[25]  John J. Leonard,et al.  Deformation-based loop closure for large scale dense RGB-D SLAM , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[26]  Ming Zeng,et al.  Templateless Quasi-rigid Shape Modeling with Implicit Loop-Closure , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[28]  Matthias Nießner,et al.  Real-time 3D reconstruction at scale using voxel hashing , 2013, ACM Trans. Graph..

[29]  Dieter Fox,et al.  Patch Volumes: Segmentation-Based Consistent Mapping with RGB-D Cameras , 2013, 2013 International Conference on 3D Vision.

[30]  Jihun Yu,et al.  Realtime facial animation with on-the-fly correctives , 2013, ACM Trans. Graph..

[31]  Jonathan T. Barron,et al.  3D self-portraits , 2013, ACM Trans. Graph..

[32]  Chen Qian,et al.  Realtime and Robust Hand Tracking from Depth , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Bo Fu,et al.  Quality Dynamic Human Body Modeling Using a Single Low-Cost Depth Camera , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Andrew W. Fitzgibbon,et al.  Real-time non-rigid reconstruction using an RGB-D camera , 2014, ACM Trans. Graph..

[35]  Dieter Fox,et al.  DART: Dense Articulated Real-Time Tracking , 2014, Robotics: Science and Systems.

[36]  Henry Fuchs,et al.  Temporally enhanced 3D capture of room-sized dynamic scenes with commodity depth cameras , 2014, 2014 IEEE Virtual Reality (VR).

[37]  Ruigang Yang,et al.  Real-Time Simultaneous Pose and Shape Estimation for Articulated Objects Using a Single Depth Camera , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.