Gestures à Go Go

Training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future data. However, recruiting participants, data collection, and labeling, etc., necessary for achieving this goal are usually time consuming and expensive. Thus, it is important to investigate how to empower developers to quickly collect gesture samples for improving UI usage and user experience. In response to this need, we introduce Gestures à Go Go (g3), a web service plus an accompanying web application for bootstrapping stroke gesture samples based on the kinematic theory of rapid human movements. The user only has to provide a gesture example once, and g3 will create a model of that gesture. Then, by introducing local and global perturbations to the model parameters, g3 generates from tens to thousands of synthetic human-like samples. Through a comprehensive evaluation, we show that synthesized gestures perform equally similar to gestures generated by human users. Ultimately, this work informs our understanding of designing better user interfaces that are driven by gestures.

[1]  Filip De Turck,et al.  WS-Gesture, a gesture-based state-aware control framework , 2010, 2010 IEEE International Conference on Service-Oriented Computing and Applications (SOCA).

[2]  T. Flash,et al.  Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. , 1995, Journal of experimental psychology. Human perception and performance.

[3]  R Plamondon,et al.  Studying the variability of handwriting patterns using the Kinematic Theory. , 2009, Human movement science.

[4]  Catherine G. Wolf,et al.  The Use of Hand-Drawn Gestures for Text Editing , 1987, Int. J. Man Mach. Stud..

[5]  R. Plamondon,et al.  A multi-level representation paradigm for handwriting stroke generation. , 2006, Human movement science.

[6]  James A. Landay,et al.  Gestalt: integrated support for implementation and analysis in machine learning , 2010, UIST.

[7]  Beryl Plimmer,et al.  A toolkit approach to sketched diagram recognition , 2007, BCS HCI.

[8]  Shumin Zhai,et al.  Using strokes as command shortcuts: cognitive benefits and toolkit support , 2009, CHI.

[9]  Anil K. Jain,et al.  Template-based online character recognition , 2001, Pattern Recognit..

[10]  David E. Meyer,et al.  Speed—Accuracy Tradeoffs in Aimed Movements: Toward a Theory of Rapid Voluntary Action , 2018, Attention and Performance XIII.

[11]  Hai Yang,et al.  ACM Transactions on Intelligent Systems and Technology - Special Section on Urban Computing , 2014 .

[12]  F. J. Maarse.  The study of handwriting movement: peripheral models and signal processing techniques , 1987 .

[13]  Sriganesh Madhvanath,et al.  Principal component analysis for online handwritten character recognition , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[14]  Jean Paul Haton,et al.  A Syntactic Approach for Handwritten Mathematical Formula Recognition , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Leclerc,et al.  4 - Des gaussiennes pour la modélisation des signatures et la segmentation de tracés manuscrits , 1992 .

[16]  A. Alimi.  Beta neuro-fuzzy systems , 2003 .

[17]  James A. Landay,et al.  SATIN: a toolkit for informal ink-based applications , 2000, UIST '00.

[18]  J. J. Denier,et al.  The guiding of human writing movements , 1965, Kybernetik.

[19]  Michele Risi,et al.  A Parsing Technique for Sketch Recognition Systems , 2004, 2004 IEEE Symposium on Visual Languages - Human Centric Computing.

[20]  Réjean Plamondon,et al.  Development of a Sigma-Lognormal representation for on-line signatures , 2009, Pattern Recognit..

[21]  Yang Li,et al.  Gesture coder: a tool for programming multi-touch gestures by demonstration , 2012, CHI.

[22]  Jean Vanderdonckt,et al.  Designing graphical user interfaces integrating gestures , 2012, SIGDOC '12.

[23]  Lisa Anthony,et al.  $N-protractor: a fast and accurate multistroke recognizer , 2012, Graphics Interface.

[24]  Reiner Marzinkewitsch.  Operating computer algebra systems by handprinted input , 1991, ISSAC '91.

[25]  Thad Starner,et al.  MAGIC summoning: towards automatic suggesting and testing of gestures with low probability of false positives during use , 2013, J. Mach. Learn. Res..

[26]  James A. Landay,et al.  Implications for a gesture design tool , 1999, CHI '99.

[27]  Yang Li,et al.  CrowdLearner: rapidly creating mobile recognizers using crowdsourcing , 2013, UIST.

[28]  Christian O'Reilly,et al.  The lognormal handwriter: learning, performing, and declining , 2013, Front. Psychol..

[29]  Daniel C. Halbert,et al.  Programming by Example , 2010, Encyclopedia of Machine Learning.

[30]  P D Neilson,et al.  The problem of redundancy in movement control: The adaptive model theory approach , 1993, Psychological research.

[31]  Julian Fiérrez,et al.  Synthetic on-line signature generation. Part II: Experimental validation , 2012, Pattern Recognit..

[32]  James Arvo,et al.  Equation entry and editing via handwriting and gesture recognition , 2001, Behav. Inf. Technol..

[33]  Olivier Bau,et al.  OctoPocus: a dynamic guide for learning gesture-based command sets , 2008, UIST '08.

[34]  Pietro Morasso,et al.  How a discontinuous mechanism can produce continuous patterns in trajectory formation and handwriting , 1983 .

[35]  Brad A. Myers,et al.  Creating user interfaces by demonstration , 1988 .

[36]  Claus Bahlmann,et al.  Online handwriting recognition with support vector machines - a kernel approach , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[37]  Lisa Anthony,et al.  A lightweight multistroke recognizer for user interface prototypes , 2010, Graphics Interface.

[38]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[39]  Yang Li,et al.  Bootstrapping personal gesture shortcuts with the wisdom of the crowd and handwriting recognition , 2012, CHI.

[40]  Tony DeRose,et al.  Proton: multitouch gestures as regular expressions , 2012, CHI.

[41]  Radu-Daniel Vatavu,et al.  Gestures as point clouds: a $P recognizer for user interface prototypes , 2012, ICMI '12.

[42]  Christian O'Reilly,et al.  Neuromuscular Representation and Synthetic Generation of Handwritten Whiteboard Notes , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[43]  Yang Li,et al.  Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes , 2007, UIST.

[44]  David Goldberg,et al.  Touch-typing with a stylus , 1993, INTERCHI.

[45]  James A. Landay,et al.  Extending an existing user interface toolkit to support gesture recognition , 1993, INTERCHI Adjunct Proceedings.

[46]  Réjean Plamondon,et al.  A kinematic theory of rapid human movements , 1995, Biological Cybernetics.

[47]  Ivan E. Sutherland,et al.  Sketchpad a Man-Machine Graphical Communication System , 1899, Outstanding Dissertations in the Computer Sciences.

[48]  Wolfgang Konen,et al.  Gesture recognition on few training data using Slow Feature Analysis and parametric bootstrap , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[49]  Réjean Plamondon,et al.  Modelling velocity profiles of rapid movements: a comparative study , 1993, Biological Cybernetics.

[50]  Atau Tanaka,et al.  Adaptive Gesture Recognition with Variation Estimation for Interactive Systems , 2014, ACM Trans. Interact. Intell. Syst..

[51]  Thad Starner,et al.  MAGIC 2.0: A web tool for false positive prediction and prevention for gesture recognition systems , 2011, Face and Gesture 2011.

[52]  Yang Li,et al.  Protractor: a fast and accurate gesture recognizer , 2010, CHI.

[53]  Réjean Plamondon,et al.  A kinematic theory of rapid human movements , 1995, Biological Cybernetics.

[54]  Radu-Daniel Vatavu,et al.  The effect of sampling rate on the performance of template-based gesture recognizers , 2011, ICMI '11.

[55]  Yang Li,et al.  Gesture studio: authoring multi-touch interactions through demonstration and declaration , 2013, CHI.

[56]  Alejandro Héctor Toselli,et al.  Context-Aware Gestures for Mixed-Initiative Text Editing UIs , 2015, Interact. Comput..

[57]  Shumin Zhai,et al.  Foundational Issues in Touch-Surface Stroke Gesture Design - An Integrative Review , 2012, Found. Trends Hum. Comput. Interact..

[58]  Wei-Tek Tsai,et al.  Gesture Profile for Web Services: An Event-Driven Architecture to Support Gestural Interfaces for Smart Environments , 2012, AmI.

[59]  Beat Signer,et al.  iGesture: A General Gesture Recognition Framework , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[60]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[61]  T. O. Ellis,et al.  The RAND tablet: a man-machine graphical communication device , 1964, AFIPS '64 (Fall, part I).

[62]  Christian O'Reilly,et al.  Synthetic Handwritten Gesture Generation Using Sigma-Lognormal Model for Evolving Handwriting Classifiers , 2010 .

[63]  J. Hollerbach.  An oscillation theory of handwriting , 2004, Biological Cybernetics.

[64]  Anind K. Dey,et al.  a CAPpella: programming by demonstration of context-aware applications , 2004, CHI.

[65]  Réjean Plamondon,et al.  Modelization of Handwriting: A System Approach , 1986 .

[66]  Manfred K. Lang,et al.  Segmentation and recognition of symbols within handwritten mathematical expressions , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[67]  Stephen Grossberg,et al.  The Vite Model: A Neural Command Circuit for Generating Arm and Articulator Trajectories, , 1988 .

[68]  Yang Li,et al.  Gesture script: recognizing gestures and their structure using rendering scripts and interactively trained parts , 2014, CHI.

[69]  Réjean Plamondon,et al.  Training of On-Line Handwriting Text Recognizers with Synthetic Text Generated Using the Kinematic Theory of Rapid Human Movements , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[70]  Thad Starner,et al.  MAGIC: a motion gesture design tool , 2010, CHI.

[71]  Tek-Jin Nam,et al.  EventHurdle: supporting designers' exploratory interaction prototyping with gesture-based sensors , 2013, CHI.

[72]  Antonio Cisternino,et al.  GestIT: a declarative and compositional framework for multiplatform gesture definition , 2013, EICS '13.

[73]  Christian O'Reilly,et al.  Recent developments in the study of rapid human movements with the kinematic theory: Applications to handwriting and signature synthesis , 2014, Pattern Recognit. Lett..

[74]  Alireza Sahami Shirazi,et al.  Understanding shortcut gestures on mobile touch devices , 2014, MobileHCI '14.