Real-Time Fingertip Tracking and Gesture Recognition

Augmented desk interfaces and other virtual reality systems depend on accurate, real-time hand and fingertip tracking for seamless integration between real objects and associated digital information. We introduce a method for discerning fingertip locations in image frames and measuring fingertip trajectories across image frames. We also propose a mechanism for combining direct manipulation and symbolic gestures based on multiple fingertip motions. Our method uses a filtering technique, in addition to detecting fingertips in each image frame, to predict fingertip locations in successive image frames and to examine the correspondences between the predicted locations and detected fingertips. This lets us obtain multiple complex fingertip trajectories in real time and improves fingertip tracking. This method can track multiple fingertips reliably even on a complex background under changing lighting conditions without invasive devices or color markers.

[1]  Yoichi Sato,et al.  Fast tracking of hands and fingertips in infrared images for augmented desk interface , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[2]  Akira Utsumi,et al.  Multiple-hand-gesture tracking using multiple cameras , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[3]  Xinlei Chen,et al.  Two-handed drawing on augmented desk system , 2002, AVI '02.

[4]  Yoichi Sato,et al.  Interactive textbook and interactive Venn diagram: natural and intuitive interfaces on augmented desk system , 2000, CHI.

[5]  Hiroshi Ishii,et al.  Illuminating light: an optical design tool with a luminous-tangible interface , 1998, CHI.

[6]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[7]  Thad Starner,et al.  Visual Recognition of American Sign Language Using Hidden Markov Models. , 1995 .

[8]  Masahiko Yachida,et al.  Multiple-human tracking using multiple cameras , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[9]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[10]  Yasuhito Suenaga,et al.  "Finger-Pointer": Pointing interface by image processing , 1994, Comput. Graph..

[11]  Jakub Segen,et al.  Shadow gestures: 3D hand pose estimation using a single camera , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[12]  Jérôme Martin,et al.  Automatic handwriting gestures recognition using hidden Markov models , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[13]  Ying Wu,et al.  Capturing natural hand articulation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[14]  Yoshiaki Shirai,et al.  Hand gesture estimation and model refinement using monocular camera-ambiguity limitation by inequality constraints , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[15]  Jakub Segen,et al.  Gesture based 3D man-machine interaction using a single camera , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[16]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[17]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  James L. Crowley,et al.  Finger Tracking as an Input Device for Augmented Reality , 1995 .

[19]  Takeo Kanade,et al.  Model-based tracking of self-occluding articulated objects , 1995, Proceedings of IEEE International Conference on Computer Vision.