Gesture Recognition by Data Fusion of Time-of-Flight and Color Cameras

In the last years numerous applications of HumanComputer Interaction have exploited the capabilities of Time-ofFlight cameras for achieving more and more comfortable and precise interactions. In particular, gesture recognition is one of the most active fields. This work presents a new method for interacting with a virtual object in a 3D space. Our approach is based on the fusion of depth data, supplied by a ToF camera, with color information, supplied by a HD webcam. The hand detection procedure does not require any learning phase and is able to concurrently manage gestures of two hands. The system is robust to the presence in the scene of other objects or people, thanks to the use of the Kalman filter for maintaining the tracking of the hands. Keywords—Gesture recognition, human-computer interaction, Time-of-Flight camera.

[1]  W. Marsden.  I and J , 2012 .

[2]  Michael Lehmann,et al.  An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (SwissRanger) , 2004, SPIE Optical Systems Design.

[3]  Thomas B. Moeslund,et al.  View invariant gesture recognition using the CSEM SwissRanger SR-2 camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[4]  Seung Wook Kim,et al.  Range Camera for Simple behind Display Interaction , 2011, MVA.

[5]  Stefan Müller,et al.  Hand Gesture Recognition with a Novel IR Time-of-Flight Range Camera-A Pilot Study , 2007, MIRAGE.

[6]  Thomas Martinetz,et al.  Deictic Gestures with a Time-of-Flight Camera , 2009, Gesture Workshop.

[7]  T. Oggier,et al.  SwissRanger SR 3000 and First Experiences based on Miniaturized 3 D-TOF Cameras , 1968 .

[8]  Joachim Hornegger,et al.  Robust real-time 3D time-of-flight based gesture navigation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[9]  Ray A. Jarvis,et al.  Visual interpretation of natural pointing gestures in 3D space for human-robot interaction , 2010, 2010 11th International Conference on Control Automation Robotics & Vision.

[10]  Luca Lombardi,et al.  Human-Computer Interaction through Time-of-Flight and RGB Cameras , 2011, ICIAP.

[11]  Ralf Reulke,et al.  Combination of distance data with high resolution images , 2006 .

[12]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[13]  S. E. Ghobadi,et al.  Hand Segmentation Using 2D/3D Images , 2007 .

[14]  Otmar Loffeld,et al.  Real Time Hand Based Robot Control Using 2D/3D Images , 2008, ISVC.

[15]  Luca Lombardi,et al.  Fast Real-time Segmentation and Tracking of Multiple Subjects by Time-of-Flight Camera - A New Approach for Real-time Multimedia Applications with 3D Camera Sensor , 2011, VISAPP.

[16]  Joachim Hornegger,et al.  Gesture recognition with a Time-Of-Flight camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[17]  Ralf Dörner,et al.  Recognition of Dynamic Hand Gestures with Time-of-Flight Cameras , 2010 .

[18]  Luc Van Gool,et al.  Combining RGB and ToF cameras for real-time 3D hand gesture interaction , 2011, WACV.