A new method for inpainting of depth maps from time-of-flight sensors based on a modified closing by reconstruction algorithm

Abstract Time-of-Flight (ToF) sensors are popular devices that extract 3D information from a scene but result to be susceptible to noise and loss of data creating holes and gaps in the boundaries of the objects. The most common approaches to tackling this problem are supported by color images with good results, however, not all ToF devices produce color information. Mathematical morphology provides operators that can manage the problem of noise in single depth frames. In this paper, a new method for the filtering of single depth maps, when no color image is available, is presented, based on a modification to the morphological closing by reconstruction algorithm. The proposed method eliminates noise, emphasizing a high contour preservation, and it is compared, both qualitative and quantitatively, with other state-of-the-art filters. The proposed method represents an improvement to the closing by reconstruction algorithm that can be applied for filter depth maps of ToF devices.

[1]  Gang Wang,et al.  Fusion of Median and Bilateral Filtering for Range Image Upsampling , 2013, IEEE Transactions on Image Processing.

[2]  Pierre Soille,et al.  Morphological Image Analysis: Principles and Applications , 2003 .

[3]  A. N. Rajagopalan,et al.  Range map superresolution-inpainting, and reconstruction from sparse data , 2012, Comput. Vis. Image Underst..

[4]  Hayde Peregrina-Barreto,et al.  Morphological rational operator for contrast enhancement. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[5]  Nabil Aouf,et al.  GPU-based real-time RGBD data filtering , 2018, Journal of Real-Time Image Processing.

[6]  Yao Wang,et al.  Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model , 2014, IEEE Transactions on Image Processing.

[7]  Sebastian Thrun,et al.  A Noise‐aware Filter for Real‐time Depth Upsampling , 2008 .

[8]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[9]  Lutz Plümer,et al.  Low-Cost 3D Systems: Suitable Tools for Plant Phenotyping , 2014, Sensors.

[10]  Eduardo Ros,et al.  Background Subtraction Based on Color and Depth Using Active Sensors , 2013, Sensors.

[11]  Iván R. Terol-Villalobos,et al.  Image segmentation and filtering based on transformations with reconstruction criteria , 2006, J. Vis. Commun. Image Represent..

[12]  Wuttipong Kumwilaisak,et al.  Optimal depth recovery using image guided TGV with depth confidence for high-quality view synthesis , 2016, J. Vis. Commun. Image Represent..

[13]  Maziar Loghman,et al.  Real-Time Refinement of Kinect Depth Maps using Multi-Resolution Anisotropic Diffusion , 2014, Mob. Networks Appl..

[14]  Carlos Sagüés,et al.  Human-Computer Interaction Based on Hand Gestures Using RGB-D Sensors , 2013, Sensors.

[15]  Atsuto Maki,et al.  Towards a simulation driven stereo vision system , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[16]  Changming Sun,et al.  Guided image completion by confidence propagation , 2016, Pattern Recognit..

[17]  Kazuhiro Fukui,et al.  Realistic CG Stereo Image Dataset with Ground Truth Disparity Maps , 2012 .

[18]  Hugues Talbot,et al.  Mathematical Morphology: from theory to applications , 2013 .

[19]  Fulvio Rinaudo,et al.  SR-4000 and CamCube3.0 Time of Flight (ToF) Cameras: Tests and Comparison , 2012, Remote. Sens..

[20]  Pao-Chi Chang,et al.  People tracking in an environment with multiple depth cameras: A skeleton-based pairwise trajectory matching scheme , 2016, J. Vis. Commun. Image Represent..

[21]  Xiaojin Gong,et al.  Guided Depth Enhancement via Anisotropic Diffusion , 2013, PCM.

[22]  Björn E. Ottersten,et al.  Unified multi-lateral filter for real-time depth map enhancement , 2015, Image Vis. Comput..

[23]  Peter A. Beling,et al.  Statistical Analysis-Based Error Models for the Microsoft Kinect™ Depth Sensor , 2014, Sensors.

[24]  Filiberto Chiabrando,et al.  Sensors for 3D Imaging: Metric Evaluation and Calibration of a CCD/CMOS Time-of-Flight Camera , 2009, Sensors.

[25]  Fan Zhong,et al.  Estimation of Kinect depth confidence through self-training , 2014, The Visual Computer.

[26]  Harry Shum,et al.  Real-Time Depth Image Acquisition and Restoration for Image Based Rendering and Processing Systems , 2015, J. Signal Process. Syst..

[27]  Dieter Fox,et al.  RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments , 2012, Int. J. Robotics Res..

[28]  Achim J. Lilienthal,et al.  Comparative evaluation of range sensor accuracy for indoor mobile robotics and automated logistics applications , 2013, Robotics Auton. Syst..

[29]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[30]  Sinan Kalkan,et al.  An iterative adaptive multi-modal stereo-vision method using mutual information , 2015, J. Vis. Commun. Image Represent..

[31]  Jean Serra,et al.  Image Analysis and Mathematical Morphology , 1983 .

[32]  Sergi Foix,et al.  Indoor and outdoor depth imaging of leaves with time-of-flight and stereo vision sensors: Analysis and comparison , 2014 .

[33]  Chun Chen,et al.  A survey of human pose estimation: The body parts parsing based methods , 2015, J. Vis. Commun. Image Represent..

[34]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[35]  Michael Schmeing,et al.  Edge-aware depth image filtering using color segmentation , 2014, Pattern Recognit. Lett..

[36]  Yaonan Wang,et al.  Obstacle Classification and 3D Measurement in Unstructured Environments Based on ToF Cameras , 2014, Sensors.

[37]  Rogério Schmidt Feris,et al.  Joint Super Resolution and Denoising From a Single Depth Image , 2015, IEEE Transactions on Multimedia.

[38]  Pablo Bustos,et al.  Model-Based Reinforcement of Kinect Depth Data for Human Motion Capture Applications , 2013, Sensors.

[39]  Minh N. Do,et al.  Depth Video Enhancement Based on Weighted Mode Filtering , 2012, IEEE Transactions on Image Processing.

[40]  Seung-Won Jung,et al.  Directional Joint Bilateral Filter for Depth Images , 2014, Sensors.

[41]  Dong Tian,et al.  Depth map processing with iterative joint multilateral filtering , 2010, 28th Picture Coding Symposium.

[42]  Francisco José Madrid-Cuevas,et al.  Simultaneous reconstruction and calibration for multi-view structured light scanning , 2016, J. Vis. Commun. Image Represent..

[43]  Mozhdeh Shahbazi,et al.  Range Camera Self-Calibration Based on Integrated Bundle Adjustment via Joint Setup with a 2D Digital Camera , 2011, Sensors.

[44]  Luc Vincent,et al.  Morphological grayscale reconstruction in image analysis: applications and efficient algorithms , 1993, IEEE Trans. Image Process..

[45]  Seung-Won Jung,et al.  Enhancement of Image and Depth Map Using Adaptive Joint Trilateral Filter , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  Qi Zhang,et al.  100+ Times Faster Weighted Median Filter (WMF) , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Rüdiger Dillmann,et al.  Autonomous grasp and manipulation planning using a ToF camera , 2012, Robotics Auton. Syst..

[48]  Ennio Gambi,et al.  A Depth-Based Fall Detection System Using a Kinect® Sensor , 2014, Sensors.

[49]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[50]  Jing Zhang,et al.  Depth map super-resolution using stereo-vision-assisted model , 2015, Neurocomputing.

[51]  Cheolkon Jung,et al.  Boundary-preserving stereo matching with certain region detection and adaptive disparity adjustment , 2015, J. Vis. Commun. Image Represent..

[52]  Jilin Liu,et al.  Guided depth enhancement via a fast marching method , 2013, Image Vis. Comput..

[53]  Manuel Graña,et al.  Tuning of Adaptive Weight Depth Map Generation Algorithms , 2012, Journal of Mathematical Imaging and Vision.