A Taxonomy of Gestures in Human Computer Interactions

We present a survey and taxonomy of gesture-based computer interactions motivated by a literature review of over 40 years of gesture based interactions. This work presents a unique perspective on gesture-based interactions, categorized in terms of four key elements: gesture styles, the application domains they are applied to, input technologies and output technologies used for implementation. The taxonomy provides a means of addressing gestures as an interaction mode across the different application domains so that researchers and designers can draw on the vast amount of research that has been addressed within the literature from an interaction perspective.

[1]  Richard A. Foulds,et al.  Gestural human-machine interaction for people with severe speech and motor impairment due to cerebral palsy , 1994, CHI Conference Companion.

[2]  Sanshzar Kettebekov Exploiting prosodic structuring of coverbal gesticulation , 2004, ICMI '04.

[3]  Pierre Wellner The DigitalDesk calculator: tangible manipulation on a desk top display , 1991, UIST '91.

[4]  Alex Pentland,et al.  The ALIVE system: wireless, full-body interaction with autonomous agents , 1997, Multimedia Systems.

[5]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[6]  Jun Rekimoto,et al.  PreSense: interaction techniques for finger sensing input devices , 2003, UIST '03.

[7]  KwangYun Wohn,et al.  The control of avatar motion using hand gesture , 1998, VRST '98.

[8]  Roger B. Dannenberg,et al.  A gesture based user interface prototyping system , 1989, UIST '89.

[9]  Jakub Segen,et al.  Video acquired gesture interfaces for the handicapped , 1998, MULTIMEDIA '98.

[10]  Steven A. Shafer,et al.  XWand: UI for intelligent spaces , 2003, CHI '03.

[11]  Rashid Ansari,et al.  Multimodal human discourse: gesture and speech , 2002, TCHI.

[12]  Tom G. Zimmerman,et al.  A hand gesture interface device , 1987, CHI '87.

[13]  Gregory D. Abowd,et al.  A gesture-based authentication scheme for untrusted public terminals , 2004, UIST '04.

[14]  David Zeltzer,et al.  A design method for “whole-hand” human-computer interaction , 1993, TOIS.

[15]  Wen Gao,et al.  Large vocabulary sign language recognition based on hierarchical decision trees , 2003, ICMI '03.

[16]  Andrew Zisserman,et al.  Vision based Interpretation of Natural Sign Languages , 2003 .

[17]  Myron W. Krueger,et al.  VIDEOPLACE—an artificial reality , 1985, CHI '85.

[18]  Warren Robinett,et al.  Virtual environment display system , 1987, I3D '86.

[19]  Kazuo Kyuma,et al.  Computer vision for computer games , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[20]  Dean Rubine,et al.  Combining gestures and direct manipulation , 1992, CHI.

[21]  Annelies Braffort A gesture recognition architecture for sign language , 1996, Assets '96.

[22]  William Buxton,et al.  Issues and techniques in touch-sensitive tablet input , 1985, SIGGRAPH '85.

[23]  Joseph A. Paradiso,et al.  Applying electric field sensing to human-computer interfaces , 1995, CHI '95.

[24]  Jakub Segen,et al.  Gesture VR: vision-based 3D hand interace for spatial interaction , 1998, MULTIMEDIA '98.

[25]  Maribeth Gandy Coleman,et al.  The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[26]  Tracy L. Westeyn,et al.  Georgia tech gesture toolkit: supporting experiments in gesture recognition , 2003, ICMI '03.

[27]  Daniel Thalmann The virtual human as a multimodal interface , 2000, AVI '00.

[28]  Mark S. Ackerman,et al.  Mediated voice communication via mobile IP , 2002, UIST '02.

[29]  Ken Hinckley,et al.  Synchronous gestures for multiple persons and computers , 2003, UIST '03.

[30]  Tovi Grossman,et al.  Multi-finger gestural interaction with 3d volumetric displays , 2004, UIST '04.

[31]  Waren Teitelman,et al.  Real time recognition of hand-drawn characters , 1964, AFIPS '64 (Fall, part I).

[32]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  John F. Canny,et al.  Cursive:: controlling expressive avatar gesture using pen gesture , 2002, CVE '02.

[34]  Loren G. Terveen,et al.  The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface , 2002, CHI Extended Abstracts.

[35]  Kristina Höök,et al.  SenToy in FantasyA: Designing an Affective Sympathetic Interface to a Computer Game , 2002, Personal and Ubiquitous Computing.

[36]  William Ribarsky,et al.  Speech and Gesture Multimodal Control of a Whole Earth 3D Visualization Environment , 2002, VisSym.

[37]  Joseph A. Paradiso,et al.  Interfacing to the foot: apparatus and applications , 2000, CHI Extended Abstracts.

[38]  J Rhyne Dialogue management for gestural interfaces , 1987, COMG.

[39]  Jun Rekimoto,et al.  Pick-and-drop: a direct manipulation technique for multiple computer environments , 1997, UIST '97.

[40]  Norbert A. Streitz,et al.  i-LAND: an interactive landscape for creativity and innovation , 1999, CHI '99.

[41]  David B. Koons,et al.  Iconic: speech and depictive gestures at the human-machine interface , 1994, CHI Conference Companion.

[42]  Hideaki Kuzuoka,et al.  GestureCam: a video communication system for sympathetic remote collaboration , 1994, CSCW '94.

[43]  Hiroaki Nishino,et al.  Interactive two-handed gesture interface in 3D virtual environments , 1997, VRST '97.

[44]  Roy Want,et al.  Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces , 1998, CHI.

[45]  Ravin Balakrishnan,et al.  Simple vs. compound mark hierarchical marking menus , 2004, UIST '04.

[46]  Kori Inkpen Quinn,et al.  That one there! Pointing to establish device identity , 2002, UIST '02.

[47]  Jan O. Borchers WorldBeat: designing a baton-based interface for an interactive music exhibit , 1997, CHI.

[48]  Richard A. Bolt,et al.  Two-handed gesture in multi-modal natural dialog , 1992, UIST '92.

[49]  Andrew S. Forsberg,et al.  UniCam—2D gestural camera controls for 3D environments , 1999, SI3D.

[50]  P. David Stotts,et al.  FaceSpace: endo- and exo-spatial hypermedia in the transparent video facetop , 2004, HYPERTEXT '04.

[51]  Hiroaki Nishino,et al.  3D object modeling using spatial and pictographic gestures , 1998, VRST '98.

[52]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[53]  TruckenbrodJoan IEEE international symposium on wearable computers , 1998 .

[54]  David Allport,et al.  Issues of gestural navigation in abstract information spaces , 1995, CHI '95.

[55]  Richard B. Reilly Applications of face and gesture recognition for human-computer interaction , 1998, MULTIMEDIA '98.

[56]  S. Kicha Ganapathy,et al.  A synthetic visual environment with hand gesturing and voice input , 1989, CHI '89.

[57]  Stephen A. Brewster,et al.  A paradigm shift: alternative interaction techniques for use with mobile & wearable devices , 2003, CASCON.

[58]  Thecla Schiphorst,et al.  Using a gestural interface toolkit for tactile input to a dynamic virtual space , 2002, CHI Extended Abstracts.

[59]  Alan Wexelblat Natural gesture in virtual environments , 1994 .

[60]  Joseph A. Paradiso Tracking contact and free gesture across large interactive surfaces , 2003, CACM.

[61]  Konrad Tollmar,et al.  Gesture + play: full-body interaction for virtual environments , 2003, CHI Extended Abstracts.

[62]  David Zeltzer,et al.  A survey of glove-based input , 1994, IEEE Computer Graphics and Applications.

[63]  Alan Wexelblat Research Challenges in Gesture: Open Issues and Unsolved Problems , 1997, Gesture Workshop.

[64]  Vladimir Pavlovic,et al.  Speech/gesture interface to a visual computing environment for molecular biologists , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[65]  Jacob Eisenstein,et al.  Visual and linguistic information in gesture classification , 2007, SIGGRAPH '07.

[66]  Jun Rekimoto,et al.  SmartSkin: an infrastructure for freehand manipulation on interactive surfaces , 2002, CHI.

[67]  Mike Wu,et al.  Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays , 2003, UIST '03.

[68]  Min Chul Kim,et al.  GIA: design of a gesture-based interaction photo album , 2004, Personal and Ubiquitous Computing.

[69]  Stephen A. Brewster,et al.  Gestural and audio metaphors as a means of control for mobile devices , 2002, CHI.

[70]  Joseph J. LaViola,et al.  Hands-free multi-scale navigation in virtual environments , 2001, I3D '01.

[71]  Rajeev Sharma,et al.  Experimental evaluation of vision and speech based multimodal interfaces , 2001, PUI '01.

[72]  Steven D. Pieper,et al.  Hands-on interaction with virtual environments , 1989, UIST '89.

[73]  Margot Brereton,et al.  Work at Hand: An Exploration of Gesture in the Context of Work and Everyday Life to Inform the Design of Gestural Input Devices , 2003, AUIC.

[74]  François Bérard,et al.  Bare-hand human-computer interaction , 2001, PUI '01.

[75]  Stéphane Chatty,et al.  Pen computing for air traffic control , 1996, CHI.

[76]  Alan Wexelblat,et al.  An approach to natural gesture in virtual environments , 1995, TCHI.

[77]  Tyson R. Henry,et al.  Integrating gesture and snapping into a user interface toolkit , 1990, UIST '90.

[78]  Monica M. C. Schraefel,et al.  The radial scroll tool: scrolling support for stylus- or touch-based document navigation , 2004, UIST '04.

[79]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[80]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[81]  Stefan Kopp,et al.  Towards integrated microplanning of language and iconic gesture for multimodal output , 2004, ICMI '04.

[82]  Francis K. H. Quek,et al.  Toward a vision-based hand gesture interface , 1994 .

[83]  Rainer Stiefelhagen,et al.  Pointing gesture recognition based on 3D-tracking of face, hands and head orientation , 2003, ICMI '03.

[84]  Xilin Chen,et al.  Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks , 2003, ICMI '03.

[85]  Randy F. Pausch,et al.  Comparing voodoo dolls and HOMER: exploring the importance of feedback in virtual environments , 2002, CHI.

[86]  Peter Robinson,et al.  The use of gestures in multimodal input , 1998, Assets '98.

[87]  Robert Pastel,et al.  Demonstrating information in simple gestures , 2004, IUI '04.

[88]  James W. Davis,et al.  A perceptual user interface for recognizing head gesture acknowledgements , 2001, PUI '01.

[89]  Wolfgang Wahlster,et al.  Combining Deictic Gestures and Natural Language for Referent Identification , 1986, COLING.

[90]  Andy Cockburn,et al.  The Design and Evaluation of a Flick Gesture for 'Back' and 'Forward' in Web Browsers , 2003, AUIC.

[91]  Xiang Cao,et al.  VisionWand: interaction techniques for large displays using a passive wand tracked in 3D , 2003, UIST '03.

[92]  Masaru Takeuchi,et al.  Description and recognition methods for sign language based on gesture components , 1997, IUI '97.

[93]  Stephen A. Brewster,et al.  Multimodal 'eyes-free' interaction techniques for wearable devices , 2003, CHI '03.

[94]  Jerry Alan Fails,et al.  Light widgets: interacting in every-day spaces , 2002, IUI '02.

[95]  Andrew S. Forsberg,et al.  The music notepad , 1998, UIST '98.

[96]  William T. Freeman,et al.  Television control by hand gestures , 1994 .

[97]  James A. Landay,et al.  Implications for a gesture design tool , 1999, CHI '99.

[98]  Brad A. Myers,et al.  A brief history of human-computer interaction technology , 1998, INTR.

[99]  Lars Bretzner,et al.  Computer Vision Based Recognition of Hand Gestures for Human-Computer Interaction , 2002 .

[100]  H. H. Avilés-Arriaga,et al.  Evaluation of a visual interface for gesticulation recognition , 2003, CLIHC '03.

[101]  Hiroshi Ishii,et al.  Bricks: laying the foundations for graspable user interfaces , 1995, CHI '95.

[102]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[103]  Joseph A. Paradiso,et al.  Sensor systems for interactive surfaces , 2000, IBM Syst. J..

[104]  Carl Gutwin,et al.  Improving interpretation of remote gestures with telepointer traces , 2002, CSCW '02.

[105]  Monica M. C. Schraefel,et al.  A study on the use of semaphoric gestures to support secondary task interactions , 2005, CHI EA '05.

[106]  Alan F. Blackwell,et al.  Dasher—a data entry interface using continuous gestures and language models , 2000, UIST '00.

[107]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[108]  Randy F. Pausch,et al.  Tailor: creating custom user interfaces based on gesture , 1990, UIST '90.

[109]  Dennis Proffitt,et al.  Two-handed virtual manipulation , 1998, TCHI.

[110]  Alexander G. Hauptmann,et al.  Speech and gestures for graphic image manipulation , 1989, CHI '89.

[111]  John R. Kender,et al.  Toward the use of gesture in traditional user interfaces , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[112]  Micah Alpern,et al.  Developing a car gesture interface for use as a secondary task , 2003, CHI Extended Abstracts.

[113]  Neff Walker,et al.  Evaluation of the CyberGlove as a whole-hand input device , 1995, TCHI.

[114]  Margaret Minsky,et al.  Manipulating simulated objects with real-world gestures using a force and position sensitive screen , 1984, SIGGRAPH.

[115]  Noritaka Osawa,et al.  Immersive graph navigation using direct manipulation and gestures , 2000, VRST '00.

[116]  Lars Bretzner,et al.  Using marking menus to develop command sets for computer vision based hand gesture interfaces , 2002, NordiCHI '02.

[117]  An empirical study of speech and gesture interaction: toward the definition of ergonomic design guidelines , 1998, CHI Conference Summary.

[118]  James L. Alty,et al.  Communicating graphical information to blind users using music: the role of context , 1998, CHI.

[119]  Giancarlo Iannizzotto,et al.  Hand tracking for human-computer interaction with Graylevel VisualGlove: turning back to the simple way , 2001, PUI '01.

[120]  C. A. Pickering Gesture recognition driver controls , 2005 .

[121]  Robert O. Ambrose,et al.  Telepresence control of the NASA/DARPA robonaut on a mobility platform , 2004, CHI.

[122]  James R. Rhyne,et al.  Gesturing with shared drawing tools , 1993, CHI '93.

[123]  James L. Crowley,et al.  Perceptual user interfaces: things that see , 2000, CACM.

[124]  Dong Hyun Jeong,et al.  Developing an efficient technique of selection and manipulation in immersive V.E. , 2000, VRST '00.

[125]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.