Moving Toward an Ecologically Valid Data Collection Protocol for 2D Gestures In Video Games

Those who design gesture recognizers and user interfaces often use data collection applications that enable users to comfortably produce gesture training samples. In contrast, games present unique contexts that impact cognitive load and have the potential to elicit rapid gesticulations as players react to dynamic conditions, which can result in high gesture form variability. However, the extent to which these gestures differ is presently unknown. To this end, we developed two games with unique mechanics, Follow the Leader (FTL) and Sleepy Town, as well as a standard data collection application. We collected gesture samples from 18 participants across all conditions for gestures of varying complexity, and through an analysis using relative, global, and distribution coverage measures, we confirm significant differences between conditions. We discuss the implications of our findings, and show that our FTL design is closer to being an ecologically valid data collection protocol with low implementation complexity.

[1]  R. Plamondon,et al.  A multi-level representation paradigm for handwriting stroke generation. , 2006, Human movement science.

[2]  Radu-Daniel Vatavu,et al.  Gestures as point clouds: a $P recognizer for user interface prototypes , 2012, ICMI '12.

[3]  RubineDean.  Specifying gestures by example , 1991 .

[4]  Joseph J. LaViola,et al.  Streamlined and accurate gesture recognition with Penny Pincher , 2016, Comput. Graph..

[5]  Radu-Daniel Vatavu,et al.  $Q: a super-quick, articulation-invariant stroke-gesture recognizer for low-resource devices , 2018, MobileHCI.

[6]  Jake K. Aggarwal,et al.  View invariant human action recognition using histograms of 3D joints , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[7]  Zhenmin Tang,et al.  Local structure based sparse representation for face recognition with single sample per person , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[8]  Radu-Daniel Vatavu,et al.  Child or Adult? Inferring Smartphone Users' Age Group from Touch Measurements Alone , 2015, INTERACT.

[9]  Sara Rosenblum,et al.  Comparing the handwriting behaviours of true and false writing with computerized handwriting measures , 2010 .

[10]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[11]  Joseph J. LaViola,et al.  A Rapid Prototyping Approach to Synthetic Data Generation for Improved 2D Gesture Recognition , 2016, UIST.

[12]  Joseph J. LaViola,et al.  Exploring the Benefits of Context in 3D Gesture Recognition for Game-Based Virtual Environments , 2015, ACM Trans. Interact. Intell. Syst..

[13]  Yang Li,et al.  Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes , 2007, UIST.

[14]  Joseph J. LaViola,et al.  Jackknife: A Reliable Recognizer with Few Samples and Many Modalities , 2017, CHI.

[15]  Dean Rubine,et al.  Specifying gestures by example , 1991, SIGGRAPH.

[16]  Radu-Daniel Vatavu,et al.  Relative accuracy measures for stroke gestures , 2013, ICMI '13.

[17]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[18]  David Pinelle,et al.  Heuristic evaluation for games: usability principles for video game design , 2008, CHI.

[19]  Joseph J. LaViola,et al.  DeepGRU: Deep Gesture Recognition Utility , 2018, ISVC.

[20]  Manfred Tscheligi,et al.  CHI '04 Extended Abstracts on Human Factors in Computing Systems , 2004, CHI 2004.

[21]  Jean Vanderdonckt,et al.  !FTL, an Articulation-Invariant Stroke Gesture Recognizer with Controllable Position, Scale, and Rotation Invariances , 2018, ICMI.

[22]  Daniel Vogel,et al.  Estimating the Perceived Difficulty of Pen Gestures , 2011, INTERACT.

[23]  Joseph J. LaViola,et al.  Wizard of Wii: toward understanding player experience in first person games with 3D gestures , 2011, FDG.

[24]  Réjean Plamondon,et al.  Training of On-Line Handwriting Text Recognizers with Synthetic Text Generated Using the Kinematic Theory of Rapid Human Movements , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[25]  Marcos Faúndez-Zanuy,et al.  EMOTHAW: A Novel Database for Emotional State Recognition From Handwriting and Drawing , 2017, IEEE Transactions on Human-Machine Systems.

[26]  Joseph J. LaViola,et al.  Penny pincher: a blazing fast, highly accurate $-family recognizer , 2015, Graphics Interface.

[27]  Réjean Plamondon,et al.  A kinematic theory of rapid human movements , 1995, Biological Cybernetics.

[28]  Radu-Daniel Vatavu,et al.  Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision , 2017, CHI.

[29]  Radu-Daniel Vatavu,et al.  Synthesizing Stroke Gestures Across User Populations: A Case for Users with Visual Impairments , 2017, CHI.

[30]  Luis A. Leiva,et al.  Gestures à Go Go , 2015, ACM Trans. Intell. Syst. Technol..

[31]  Joseph J. LaViola,et al.  Exploring the Trade-off Between Accuracy and Observational Latency in Action Recognition , 2013, International Journal of Computer Vision.

[32]  Javier Hernandez-Ortega,et al.  Predicting Age Groups from Touch Patterns based on Neuromotor Models , 2017 .

[33]  Gang Wang,et al.  NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Heather Desurvire,et al.  Using heuristics to evaluate the playability of games , 2004, CHI EA '04.

[35]  Steven W. Smith,et al.  The Scientist and Engineer's Guide to Digital Signal Processing , 1997 .

[36]  Shumin Zhai,et al.  Modeling human performance of pen stroke gestures , 2007, CHI.

[37]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.

[38]  Anil K. Jain,et al.  A modified Hausdorff distance for object matching , 1994, Proceedings of 12th International Conference on Pattern Recognition.