Lightweight palm and finger tracking for real-time 3D gesture control

We present a novel technique implementing barehanded interaction with virtual 3D content by employing a time-of-flight camera. The system improves on existing 3D multi-touch systems by working regardless of lighting conditions and supplying a working volume large enough for multiple users. Previous systems were limited either by environmental requirements, working volume, or computational resources necessary for realtime operation. By employing a time-of-flight camera, the system is capable of reliably recognizing gestures at the finger level in real-time at more than 50 fps with commodity computer hardware using our newly developed precision hand and finger-tracking algorithm. Building on this algorithm, the system performs gesture recognition with simple constraint modeling over statistical aggregations of the hand appearances in a working volume of more than 8 cubic meters. Two iterations of user tests were performed on a prototype system, demonstrating the feasibility and usability of the approach as well as providing first insights regarding the acceptance of true barehanded touch-based 3D interaction.

[1]  R. Lange,et al.  Solid-state time-of-flight range camera , 2001 .

[2]  Hye-Jin Kim,et al.  AR-KLT based Hand Tracking , 2006, ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication.

[3]  Horst Bischof,et al.  Real time appearance based hand tracking , 2008, 2008 19th International Conference on Pattern Recognition.

[4]  Shan Lu,et al.  Using multiple cues for hand tracking and model refinement , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[5]  Yang Liu,et al.  A robust hand tracking and gesture recognition method for wearable visual interfaces and its applications , 2004, Third International Conference on Image and Graphics (ICIG'04).

[6]  Stan Sclaroff,et al.  3D hand pose reconstruction using specialized mappings , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  Santanu Chaudhury,et al.  Robust shape based two hand tracker , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[8]  Mathias Kölsch,et al.  Robust hand detection , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[9]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[10]  Fuchun Sun,et al.  Visual Hand Tracking Using MDSA Method , 2006, The Proceedings of the Multiconference on "Computational Engineering in Systems Applications".

[11]  Kanad K. Biswas,et al.  Real Time Hand Tracking and Gesture Recognition , 2009, IPCV.

[12]  Guozhong Dai,et al.  Tracking of deformable human hand in real time as continuous input for gesture-based interaction , 2007, IUI '07.

[13]  Gordon Cheng,et al.  Unconstrained Real-time Markerless Hand Tracking for Humanoid Interaction , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[14]  Tom Drummond,et al.  Faster and Better: A Machine Learning Approach to Corner Detection , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Otmar Loffeld,et al.  Real Time Hand Based Robot Control Using 2D/3D Images , 2008, ISVC.

[16]  Clayton Lewis,et al.  Designing for usability—key principles and what designers think , 1983, CHI '83.

[17]  Jakub Segen,et al.  Gesture VR: vision-based 3D hand interace for spatial interaction , 1998, MULTIMEDIA '98.

[18]  Carmelo Ardito,et al.  Extending large-scale event participation with user-created mobile media on a public display , 2007, MUM.

[19]  Rasmus Larsen,et al.  Model-Based Hand Gesture Tracking in ToF Image Sequences , 2010, AMDO.

[20]  Mathias Kölsch,et al.  Hand Tracking with Flocks of Features , 2005, CVPR.

[21]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[22]  Olivier D. Faugeras,et al.  Finding pose of hand in video images: a stereo-based approach , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[23]  Yoichi Sato,et al.  Real-time input of 3D pose and gestures of a user's hand and its applications for HCI , 2001, Proceedings IEEE Virtual Reality 2001.

[24]  Grenoble Cede,et al.  Visual Tracking of Bare Fingers for Interactive Surfaces , 2004 .

[25]  Jakob Nielsen,et al.  The usability engineering life cycle , 1992, Computer.

[26]  J. P. Lewis,et al.  SmartCanvas: a gesture-driven intelligent drawing desk system , 2005, IUI.

[27]  Chris Joslin,et al.  A Framework for 3D Hand Tracking and Gesture Recognition using Elements of Genetic Programming , 2007, Fourth Canadian Conference on Computer and Robot Vision (CRV '07).

[28]  Tsukasa Ogasawara,et al.  A hand-pose estimation for vision-based human interfaces , 2003, IEEE Trans. Ind. Electron..

[29]  Hideo Saito,et al.  3-D drawing system via hand motion recognition from two cameras , 2000, Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cybernetics. 'cybernetics evolving to systems, humans, organizations, and their complex interactions' (cat. no.0.

[30]  Josh Wall.  Demo I Microsoft Surface and the Single View Platform , 2009 .

[31]  Manolis I. A. Lourakis,et al.  Tracking Multiple Colored Blobs with a Moving Camera , 2005, CVPR.