Fast Real-time Segmentation and Tracking of Multiple Subjects by Time-of-Flight Camera - A New Approach for Real-time Multimedia Applications with 3D Camera Sensor

Time-of-Flight cameras are a new kind of sensors that use near-infrared light to provide distance measures of an environment. In this paper we present a very fast method for real-time segmentation and tracking, that exploits the peculiar characteristics of these devices. The foreground segmentation is achieved by a dynamic thresholding and region growing: an appropriate correction based on flexible intensity thresholding and mathematical morphology is used to partially compensate one of the most common problem of the TOF cameras, the noise generated by sun light. By the use of a Kalman filter for tracking the retrieved objects the system is able to correctly handle the occlusions and to follow multiple objects placed at different distances. The proposed system is our basic step for complex multimedia applications, such as augmented reality. An example of mixed reality that includes the integration of color information, supplied by a webcam is shown in the experimental results.

[1]  D. Falie,et al.  Measurements with ToF Cameras and Their Necessary Corrections , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[2]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[3]  Rasmus Larsen,et al.  Cluster tracking with Time-of-Flight cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[4]  Q. M. Jonathan Wu,et al.  Multiple Object Tracking Based on Adaptive Depth Segmentation , 2008, 2008 Canadian Conference on Computer and Robot Vision.

[5]  Reinhard Koch,et al.  Integration of a Time-of-Flight Camera into a Mixed Reality System for Handling Dynamic Scenes, Moving Viewpoints and Occlusions in Real-Time , 2008 .

[6]  Ryan Crabb,et al.  Real-time foreground segmentation via range and color imaging , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[7]  Ralf Reulke,et al.  Combination of distance data with high resolution images , 2006 .

[8]  Luigi di Stefano,et al.  People Tracking Using a Time-of-Flight Depth Sensor , 2006, 2006 IEEE International Conference on Video and Signal Based Surveillance.

[9]  Luca Lombardi,et al.  Evaluation of a Foreground Segmentation Algorithm for 3D Camera Sensors , 2009, ICIAP.

[10]  Michael Lehmann,et al.  An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (SwissRanger) , 2004, SPIE Optical Systems Design.

[11]  Rasmus Larsen,et al.  TOF imaging in Smart room environments towards improved people tracking , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[12]  K. Hartmann,et al.  Data-Fusion of PMD-Based Distance-Information and High-Resolution RGB-Images , 2007, 2007 International Symposium on Signals, Circuits and Systems.

[13]  Raúl Rojas,et al.  High resolution segmentation with a time-of-flight 3D-camera using the example of a lecture scene , 2006 .