Foundational Issues in Touch-Surface Stroke Gesture Design - An Integrative Review

The potential for using stroke gestures to enter, retrieve and select commands and text has been recently unleashed by the popularity of touchscreen devices. This monograph provides a state-of-the-art integrative review of a body of human–computer interaction research on stroke gestures. It begins with an analysis of the design dimensions of stroke gestures as an interaction medium. The analysis classifies gestures into analogue versus abstract gestures, gestures for commands versus for symbols, gestures with different orders of complexity, visual-spatial dependent and independent gestures, and finger versus stylus drawn gestures. Gesture interfaces such as the iOS interface, the Graffiti text entry method for Palm devices, marking menus, and the SHARK/ShapeWriter word-gesture keyboard, make different choices in this multi-dimensional design space. The main body of this work consists of reviewing and synthesizing some of the foundational studies in the literature on stroke gesture interaction, particularly those done by the authors in the last decade. The human performance factors covered include motor control complexity, visual and auditory feedback, and human memory capabilities in dealing with gestures. Based on these foundational studies this review presents a set of design principles for creating stroke gesture interfaces. These include making gestures analogous to physical effects or cultural conventions, keeping gestures simple and distinct, defining stroke gestures systematically, making them self-revealing, supporting appropriate levels of chunking, and facilitating progress from visually guided performance to recall-driven performance. The overall theme is on making learning gestures easier while designing for long-term efficiency. Important system implementation issues of stroke gesture interfaces such as gesture recognition algorithms and gesture design toolkits are also covered in this review. The monograph ends with a few call-to-action research topics.

[1]  Shumin Zhai,et al.  Human Action Laws in Electronic Virtual Worlds: An Empirical Study of Path Steering Performance in VR , 2004, Presence: Teleoperators & Virtual Environments.

[2]  Herbert Freeman,et al.  On the Encoding of Arbitrary Geometric Configurations , 1961, IRE Trans. Electron. Comput..

[3]  Shumin Zhai,et al.  A comparative evaluation of finger and pen stroke gestures , 2012, CHI.

[4]  Patrick Baudisch,et al.  Understanding touch , 2011, CHI.

[5]  Olivier Bau,et al.  Scale detection for a priori gesture recognition , 2010, CHI.

[6]  Bay-Wei Chang,et al.  Animation: from cartoons to the user interface , 1993, UIST '93.

[7]  Jason Hong,et al.  Computational Support for Sketching in Design: A Review , 2009, Found. Trends Hum. Comput. Interact..

[8]  James R. Rhyne,et al.  Gesturing with shared drawing tools , 1993, CHI '93.

[9]  Ravin Balakrishnan,et al.  Pressure widgets , 2004, CHI.

[10]  M. S. Mayzner,et al.  Tables of single-letter and digram frequency counts for various word-length and letter-position combinations. , 1965 .

[11]  David G. Stork,et al.  Pattern Classification , 1973 .

[12]  William Buxton,et al.  User learning and performance with marking menus , 1994, CHI 1994.

[13]  Palmer Morrel-Samuels,et al.  Clarifying the Distinction Between Lexical and Gestural Commands , 1990, Int. J. Man Mach. Stud..

[14]  A. Baddeley.  Human Memory: Theory and Practice, Revised Edition , 1990 .

[15]  Shumin Zhai,et al.  Scale effects in steering law tasks , 2001, CHI.

[16]  Thomas P. Moran,et al.  Tivoli: an electronic whiteboard for informal workgroup meetings , 1993, INTERCHI.

[17]  Shumin Zhai,et al.  Movement model, hits distribution and learning in virtual keyboarding , 2002, CHI.

[18]  D. Norman.  Emotional design : why we love (or hate) everyday things , 2004 .

[19]  William Buxton,et al.  Contextual Animation of Gestural Commands , 1994, Comput. Graph. Forum.

[20]  Yang Li,et al.  Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes , 2007, UIST.

[21]  William Buxton,et al.  A three-state model of graphical input , 1990, INTERACT.

[22]  David Goldberg,et al.  Touch-typing with a stylus , 1993, INTERCHI.

[23]  P J Keuss,et al.  The role of vision in the temporal and spatial control of handwriting. , 1992, Acta psychologica.

[24]  Jerry Kaplan,et al.  Startup: A Silicon Valley Adventure , 1995 .

[25]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[26]  Shumin Zhai,et al.  Improving word-recognizers using an interactive lexicon with active and passive words , 2008, IUI '08.

[27]  M. Weiser,et al.  An empirical comparison of pie vs. linear menus , 1988, CHI '88.

[28]  Caroline Appert,et al.  SwingStates: adding state machines to Java and the Swing toolkit , 2008, Softw. Pract. Exp..

[29]  Joseph J. LaViola,et al.  GestureBar: improving the approachability of gesture-based interfaces , 2009, CHI.

[30]  R. Shiffrin,et al.  Controlled and automatic human information processing: I , 1977 .

[31]  Yang Li,et al.  Protractor: a fast and accurate gesture recognizer , 2010, CHI.

[32]  Mary Czerwinski,et al.  Beyond the Desktop Metaphor in Seven Dimensions , 2007 .

[33]  Shumin Zhai,et al.  Beyond Fitts' law: models for trajectory-based HCI tasks , 1997, CHI Extended Abstracts.

[34]  Charles C. Tappert,et al.  Cursive Script Recognition by Elastic Matching , 1982, IBM J. Res. Dev..

[35]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[36]  Xiang Cao,et al.  Detecting and leveraging finger orientation for interaction with direct-touch surfaces , 2009, UIST '09.

[37]  Shumin Zhai,et al.  Performance Optimization of Virtual Keyboards , 2002, Hum. Comput. Interact..

[38]  Shumin Zhai,et al.  Performance evaluation of input devices in trajectory-based tasks: an application of the steering law , 1999, CHI '99.

[39]  Olivier Bau,et al.  OctoPocus: a dynamic guide for learning gesture-based command sets , 2008, UIST '08.

[40]  T. Flash,et al.  Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. , 1995, Journal of experimental psychology. Human perception and performance.

[41]  Stéphane Chatty,et al.  Pen computing for air traffic control , 1996, CHI.

[42]  Akira Nakanishi,et al.  Writing Systems of the World , 1980 .

[43]  William Buxton,et al.  Issues in combining marking and direct manipulation techniques , 1991, UIST '91.

[44]  Brad A. Myers,et al.  EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion , 2003, UIST '03.

[45]  Martin R. Gibbs,et al.  Mediating intimacy: designing technologies to support strong-tie relationships , 2005, CHI.

[46]  William Buxton,et al.  Manual deskterity: an exploration of simultaneous pen + touch direct input , 2010, CHI EA '10.

[47]  Shumin Zhai,et al.  Shorthand writing on stylus keyboard , 2003, CHI '03.

[48]  Stephen A. Brewster,et al.  Gestural and audio metaphors as a means of control for mobile devices , 2002, CHI.

[49]  Shumin Zhai,et al.  Performance Optimization of Virtual Keyboards , 2002 .

[50]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[51]  Shumin Zhai,et al.  Modeling human performance of pen stroke gestures , 2007, CHI.

[52]  Shumin Zhai,et al.  Manual and cognitive benefits of two-handed input: an experimental study , 1998, TCHI.

[53]  Beat Signer,et al.  iGesture: A Java framework for the development and deployment of stoke-based online Gesture recognition algorithms , 2011 .

[54]  Bill Buxton,et al.  GEdit: a test bed for editing by contiguous gestures , 1991, SGCH.

[55]  Shumin Zhai,et al.  Command strokes with and without preview: using pen gestures on keyboard for command selection , 2007, CHI.

[56]  J. Gibson.  The Ecological Approach to Visual Perception , 1979 .

[57]  Catherine G. Wolf,et al.  The Use of Hand-Drawn Gestures for Text Editing , 1987, Int. J. Man Mach. Stud..

[58]  James Arvo,et al.  Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes , 2000, UIST '00.

[59]  Jakob Nielsen,et al.  Gestural interfaces: a step backward in usability , 2010, INTR.

[60]  Desney S. Tan,et al.  InkSeine: In Situ search for active note taking , 2007, CHI.

[61]  R. Johansson,et al.  Hand Movements , 2001 .

[62]  F. Craik,et al.  Levels of Pro-cessing: A Framework for Memory Research , 1975 .

[63]  Pierre Dragicevic,et al.  Earpod: eyes-free menu selection using touch input and reactive audio feedback , 2007, CHI.

[64]  Brad A. Myers,et al.  In-stroke word completion , 2006, UIST.

[65]  Thomas Grechenig,et al.  Comparing the immediate usability of graffiti 2 and virtual keyboard , 2004, CHI EA '04.

[66]  Abigail Sellen,et al.  An Empirical Evaluation of Some Articulatory and Cognitive Aspects of Marking Menus , 1993, Hum. Comput. Interact..

[67]  Ching Y. Suen,et al.  The State of the Art in Online Handwriting Recognition , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[68]  Radu-Daniel Vatavu,et al.  The effect of sampling rate on the performance of template-based gesture recognizers , 2011, ICMI '11.

[69]  William Buxton,et al.  Chunking and Phrasing and the Design of Human-Computer Dialogues (Invited Paper) , 1995, IFIP Congress.

[70]  Alan D. Baddeley,et al.  Human Memory: Theory and Practice, Revised Edition , 1990 .

[71]  Brad A. Myers,et al.  Gestural text entry on multiple devices , 2005, Assets '05.

[72]  Robert F. Sproull,et al.  Principles in interactive computer graphics , 1973 .

[73]  Gregory D. Abowd,et al.  Cirrin: a word-level unistroke keyboard for pen input , 1998, UIST '98.

[74]  Pierre Dragicevic,et al.  Strategies for accelerating on-line learning of hotkeys , 2007, CHI.

[75]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[76]  Shumin Zhai,et al.  “Writing with music”: Exploring the use of auditory feedback in gesture interfaces , 2008, TAP.

[77]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[78]  James A. Landay,et al.  SATIN: a toolkit for informal ink-based applications , 2006, SIGGRAPH Courses.

[79]  Henry Rogers,et al.  The Writing Systems of the World , 1991 .

[80]  George W. Fitzmaurice,et al.  The Hotbox: efficient access to a large number of menu-items , 1999, CHI '99.

[81]  M. Smyth,et al.  Functions of vision in the control of handwriting , 1987 .

[82]  James A. Landay,et al.  Visual similarity of pen gestures , 2000, CHI.

[83]  Ricardo Gutierrez-Osuna,et al.  GestureCommander: continuous touch-based gesture prediction , 2012, CHI Extended Abstracts.

[84]  François Guimbretière,et al.  Techniques , 2011, Laboratory Investigation.

[85]  Eric Clarke,et al.  Rhythm and Timing in Music , 1999 .

[86]  Per Ola Kristensson,et al.  Discrete and Continuous Shape Writing for Text Entry and Control , 2007 .

[87]  E. R. Crossman.  A THEORY OF THE ACQUISITION OF SPEED-SKILL∗ , 1959 .

[88]  William Buxton,et al.  Pen + touch = new tools , 2010, UIST.

[89]  Poika Isokoski,et al.  Model for unistroke writing time , 2001, CHI.

[90]  Donald A. Norman,et al.  Emotional design , 2004, UBIQ.

[91]  Shumin Zhai,et al.  CHAPTER 7 – Introduction to Shape Writing , 2007 .

[92]  I. Scott MacKenzie,et al.  Graffiti vs. unistrokes: an empirical comparison , 2008, CHI.

[93]  T. L. Dimond,et al.  Devices for reading handwritten characters , 1899, IRE-ACM-AIEE '57 (Eastern).

[94]  William Buxton,et al.  Usability evaluation considered harmful (some of the time) , 2008, CHI.

[95]  Walter Schneider,et al.  Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. , 1977 .

[96]  Lisa Anthony,et al.  A lightweight multistroke recognizer for user interface prototypes , 2010, Graphics Interface.

[97]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[98]  Eric Saund,et al.  Sloppy selection: Providing an accurate interpretation of imprecise selection gestures , 2005, Comput. Graph..

[99]  Ching,et al.  The State of the Art in On-Line Handwriting Recognition , 2000 .

[100]  R. Schmidt,et al.  New Conceptualizations of Practice: Common Principles in Three Paradigms Suggest New Concepts for Training , 1992 .

[101]  M. Gazzaniga,et al.  The new cognitive neurosciences , 2000 .

[102]  Per Ola Kristensson,et al.  Continuous Recognition and Visualization of Pen Strokes and Touch-Screen Gestures , 2011, SBIM.

[103]  C. Shea,et al.  Pew (1966) Revisited: Acquisition of Hierarchical Control as a Function of Observational Practice. , 1992, Journal of motor behavior.

[104]  Brad A. Myers,et al.  Trackball text entry for people with motor impairments , 2006, CHI.

[105]  Mark L. Johnson.  The body in the mind: the bodily basis of meaning , 1987 .

[106]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[107]  Shumin Zhai,et al.  SHARK2: a large vocabulary shorthand writing system for pen-based computers , 2004, UIST '04.

[108]  Yang Li,et al.  Experimental analysis of mode switching techniques in pen-based user interfaces , 2005, CHI.

[109]  Don Hopkins,et al.  The design and implementation of pie menus , 1991 .

[110]  ZhaiShumin,et al.  Writing with music , 2008 .

[111]  P. Viviani,et al.  Trajectory determines movement dynamics , 1982, Neuroscience.

[112]  Ali Israr,et al.  TeslaTouch: electrovibration for touch surfaces , 2010, UIST.

[113]  Shumin Zhai,et al.  The performance of touch screen soft buttons , 2009, CHI.

[114]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[115]  James A. Landay,et al.  Implications for a gesture design tool , 1999, CHI '99.

[116]  R. Proctor,et al.  Skill acquisition and human performance , 1995 .

[117]  Catherine G. Wolf.  CAN PEOPLE USE GESTURE COMMANDS? , 1986, SGCH.

[118]  William Buxton,et al.  Issues and techniques in touch-sensitive tablet input , 1985, SIGGRAPH '85.

[119]  Shumin Zhai,et al.  Using strokes as command shortcuts: cognitive benefits and toolkit support , 2009, CHI.

[120]  Shumin Zhai,et al.  The word-gesture keyboard: reimagining keyboard interaction , 2012, CACM.

[121]  Johnny Accot,et al.  Les tâches trajectorielles en interaction homme-machine : cas des tâches de navigation , 2001 .

[122]  Dan R. Olsen,et al.  Evaluating user interface systems research , 2007, UIST.

[123]  Yang Li.  Gesture search: a tool for fast mobile data access , 2010, UIST '10.

[124]  Shumin Zhai,et al.  Hard lessons: effort-inducing interfaces benefit spatial learning , 2007, CHI.

[125]  T. Moran,et al.  Beyond the Desktop Metaphor in Seven Dimensions , 2006 .

[126]  Shumin Zhai,et al.  Pen pressure control in trajectory-based interaction , 2010, Behav. Inf. Technol..

[127]  Ravin Balakrishnan,et al.  Simple vs. compound mark hierarchical marking menus , 2004, UIST '04.

[128]  John D. Gould,et al.  Behavioral experiments on handmarkings , 1987, TOIS.

[129]  Shumin Zhai,et al.  More than dotting the i's --- foundations for crossing-based interfaces , 2002, CHI.

[130]  James A. Landay,et al.  Extending an existing user interface toolkit to support gesture recognition , 1993, INTERCHI Adjunct Proceedings.

[131]  Daniel B. Willingham,et al.  A Neuropsychological Theory of Motor Skill Learning , 2004 .

[132]  Dan Venolia,et al.  T-Cube: a fast, self-disclosing pen-based alphabet , 1994, CHI '94.

[133]  James A. Landay,et al.  Interactive sketching for the early stages of user interface design , 1995, CHI '95.

[134]  Shumin Zhai,et al.  Shapewriter on the iphone: from the laboratory to the real world , 2009, CHI Extended Abstracts.

[135]  Robert A. Bjork,et al.  Optimum rehearsal patterns and name learning , 1978 .

[136]  Tyson R. Henry,et al.  Integrating gesture and snapping into a user interface toolkit , 1990, UIST '90.

[137]  I. Scott MacKenzie,et al.  The Immediate Usability of Graffiti , 1997, Graphics Interface.

[138]  Patrick Baudisch,et al.  Design and analysis of delimiters for selection-action pen gesture phrases in scriboli , 2005, CHI.

[139]  Stan Liebowitz.  Typing Errors , 2008 .

[140]  P. David.  Clio and the Economics of QWERTY , 1985 .

[141]  James A. Landay,et al.  "Those look similar!" issues in automating gesture design advice , 2001, PUI '01.

[142]  Ken Perlin,et al.  Quikwriting: continuous stylus-based text entry , 1998, UIST '98.

[143]  Stan J. Liebowitz,et al.  The Fable of the Keys , 1990, The Journal of Law and Economics.

[144]  Shumin Zhai,et al.  Learning shape writing by game playing , 2007, CHI Extended Abstracts.

[145]  Mark W. Newman,et al.  DENIM: An Informal Web Site Design Tool Inspired by Observations of Practice , 2003, Hum. Comput. Interact..