Visual gesture interfaces for virtual environments

Abstract Virtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual objects. Unfortunately, techniques for interacting with such environments are yet to mature. Gloves and sensor-based trackers are unwieldy, constraining and uncomfortable to use. A natural, more intuitive method of interaction would be to allow the user to grasp objects with their hands and manipulate them as if they were real objects. We are investigating the use of computer vision in implementing a natural interface based on hand gestures. A framework for a gesture recognition system is introduced along with results of experiments in colour segmentation, feature extraction and template matching for finger and hand tracking, and simple hand pose recognition. Implementation of a gesture interface for navigation and object manipulation in virtual environments is presented.

[1]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[2]  David C. Hogg,et al.  3D Deformable Hand Models , 1996, Gesture Workshop.

[3]  S. Chatterjee,et al.  Regression Analysis by Example , 1979 .

[4]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .

[5]  Kaisa Väänänen,et al.  Gesture Driven Interaction as a Human Factor in Virtual Environments - An Approach with Neural Networks , 1993, Virtual Reality Systems.

[6]  Rómer Rosales.  Recognition of Human Action Using Moment-Based Features , 1998 .

[7]  Alex T. Pang,et al.  Cutting planes and beyond , 1997, Comput. Graph..

[8]  José Teixeira,et al.  Advanced interaction techniques in virtual environments , 1993, Comput. Graph..

[9]  Geoffrey E. Hinton,et al.  Glove-TalkII-a neural-network interface which maps gestures to parallel formant speech synthesizer controls , 1997, IEEE Trans. Neural Networks.

[10]  Christopher R. Wren,et al.  Real-Time 3-D Tracking of the Human Body , 1996 .

[11]  Chris Hand,et al.  A Survey of 3D Interaction Techniques , 1997, Comput. Graph. Forum.

[12]  Kevin Curran,et al.  A paradigm shift in the distribution of multimedia , 2000, CACM.

[13]  Alex Pentland,et al.  Understanding purposeful human motion , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[14]  Doug A. Bowman,et al.  Formalizing the Design, Evaluation, and Application of Interaction Techniques for Immersive Virtual Environments , 1999, J. Vis. Lang. Comput..

[15]  Robert J. Schalkoff,et al.  Pattern recognition - statistical, structural and neural approaches , 1991 .

[16]  Vladimir Pavlovic,et al.  Speech/Gesture Interface to a Visual-Computing Environment , 2000, IEEE Computer Graphics and Applications.

[17]  Christopher Richard Wren,et al.  Understanding expressive action , 2000 .

[18]  Allen Avner,et al.  Interacting with computer users: Design considerations , 1980 .

[19]  Jakub Segen,et al.  Gesture VR: vision-based 3D hand interace for spatial interaction , 1998, MULTIMEDIA '98.

[20]  John Kenneth Salisbury,et al.  Making graphics physically tangible , 1999, CACM.

[21]  Jakub Segen,et al.  Look ma, no mouse! , 2000, CACM.

[22]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[23]  Yoshiaki Shirai,et al.  Hand gesture estimation and model refinement using monocular camera-ambiguity limitation by inequality constraints , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[24]  Alexander H. Waibel,et al.  A real-time face tracker , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[25]  M. R. J. Kohler,et al.  Video Based Gesture Recognition for Human Computer Interaction , 1995 .

[26]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Other Conferences.

[27]  Vladimir Pavlovic,et al.  Gestural interface to a visual computing environment for molecular biologists , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[28]  David Alan Becker,et al.  Sensei, a real-time recognition, feedback and training system for T'ai chi gestures , 1997 .

[29]  Takeo Kanade,et al.  DigitEyes: Vision-Based Human Hand Tracking , 1993 .