Reconstruction of Deformation from Depth and Color Video with Explicit Noise Models

Depth sensors like ToF cameras and structured light devices provide valuable scene information, but do not provide a stable base for optical flow or feature movement calculation because the lack of texture information makes depth image registration very challenging. Approaches associating depth values with optical flow or feature movement from color images try to circumvent this problem, but suffer from the fact that color features are often generated at edges and depth discontinuities, areas in which depth sensors inherently deliver unstable data. Using deformation tracking as an application, this article will discuss the benefits of Analysis by Synthesis (AbS) while approaching the tracking problem and how it can be used to:

[1]  Pedro Larrañaga,et al.  Towards a New Evolutionary Computation - Advances in the Estimation of Distribution Algorithms , 2006, Towards a New Evolutionary Computation.

[2]  R. Koch,et al.  CALIBRATION OF A PMD-CAMERA USING A PLANAR CALIBRATION PATTERN TOGETHER WITH A MULTI-CAMERA SETUP , 2008 .

[3]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[4]  Tim Weyrich,et al.  Capturing Time-of-Flight data with confidence , 2011, CVPR 2011.

[5]  Igor Guskov,et al.  3D object recognition from range images using pyramid matching , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[6]  C Tomasi,et al.  Shape and motion from image streams: a factorization method. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[8]  Reinhard Koch,et al.  Time-of-Flight sensor calibration for accurate range sensing , 2010, Comput. Vis. Image Underst..

[9]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[10]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  D. Falie,et al.  Noise Characteristics of 3D Time-of-Flight Cameras , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[12]  Thomas W. Parks,et al.  Adaptive homogeneity-directed demosaicing algorithm , 2005, IEEE Transactions on Image Processing.

[13]  L. Piegl,et al.  The NURBS Book , 1995, Monographs in Visual Communications.

[14]  Giancarlo Calvagno,et al.  Demosaicing With Directional Filtering and a posteriori Decision , 2007, IEEE Transactions on Image Processing.

[15]  Pattie Maes,et al.  Flexpad: highly flexible bending interactions for projected handheld displays , 2013, CHI.

[16]  Les A. Piegl,et al.  The NURBS book (2nd ed.) , 1997 .

[17]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[18]  R. Reulke,et al.  Remote Sensing and Spatial Information Sciences , 2005 .

[19]  Reinhard Koch,et al.  Direct Model-Based Tracking of 3D Object Deformations in Depth and Color Video , 2012, International Journal of Computer Vision.

[20]  Reinhard Koch,et al.  Dynamic 3-D Scene Analysis Through Synthesis Feedback Control , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Andrew E. Johnson,et al.  Spin-Images: A Representation for 3-D Surface Matching , 1997 .

[22]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[23]  Reinhard Koch,et al.  Fast Tracking of Deformable Objects in Depth and Colour Video , 2011, BMVC.

[24]  Reinhard Koch,et al.  Simultaneous Estimation of Material Properties and Pose for Deformable Objects from Depth and Color Images , 2012, DAGM/OAGM Symposium.

[25]  John G. Harris,et al.  Rigid body motion from range image sequences , 1991, CVGIP Image Underst..

[26]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[27]  Andrew Zisserman,et al.  Multiple View Geometry , 1999 .

[28]  Joachim Hornegger,et al.  Assessment of Time-of-Flight Cameras for Short Camera-Object Distances , 2011, VMV.

[29]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Comparing Review , 2006, Towards a New Evolutionary Computation.

[30]  Dana H. Ballard,et al.  Computer Vision , 1982 .