Eliciting Pen-Holding Postures for General Input with Suitability for EMG Armband Detection

We conduct a two-part study to better understand pen grip postures for general input like mode switching and com-mand invocation. The first part of the study asks participants what variations of their normal pen grip posture they might use, without any specific consideration for sensing capabilities. The second part evaluates three of their sug-gested postures with an additional set of six postures designed for the sensing capabilities of a consumer EMG armband. Results show that grips considered normal and mature, such as the dynamic tripod and the dynamic quadrupod, are the best candidates for pen-grip based interaction, followed by finger-on-pen postures and grips using pen tilt. A convolutional neural network trained on EMG data gathered during the study yields above 70% within-participant recognition accuracy for common sets of five postures and above 80% for three-posture subsets. Based on the results, we propose design guidelines for pen interaction using variations of grip postures.

[1]  Fei Su,et al.  Sensing Posture-Aware Pen+Touch Interaction on Tablets , 2019, CHI.

[2]  Parth H. Pathak,et al.  Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch , 2015, HotMobile.

[3]  Mike Fraser,et al.  SensIR: Detecting Hand Gestures with a Wearable Bracelet using Infrared Transmission and Reflection , 2017, UIST.

[4]  Joseph A. Paradiso,et al.  WristFlex: low-power gesture input with wrist-worn pressure sensors , 2014, UIST.

[5]  Audrey Girouard,et al.  FlexStylus: Leveraging Bend Input for Pen Interaction , 2017, UIST.

[6]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[7]  Clément Gosselin,et al.  Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[8]  Ethan V. Munson,et al.  Is 100 Milliseconds Too Fast? , 2001, CHI Extended Abstracts.

[9]  Heather Carnahan,et al.  Effect of pencil grasp on the speed and legibility of handwriting in children. , 2012, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[10]  Andruid Kerne,et al.  High-performance pen + touch modality interactions: a real-time strategy game eSports context , 2012, UIST.

[11]  Xiang Cao,et al.  Enhancing naturalness of pen-and-tablet drawing through context sensing , 2011, ITS '11.

[12]  Jiro Tanaka,et al.  Interaction Technique for a Pen-Based Interface Using Finger Motions , 2009, HCI.

[13]  Fabrice Matulic,et al.  Pen and touch gestural environment for document editing on interactive tabletops , 2013, ITS.

[14]  Antti Oulasvirta,et al.  Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  Daniel Vogel,et al.  Direct Pen Interaction With a Conventional Graphical User Interface , 2010, Hum. Comput. Interact..

[16]  Fabrice Matulic,et al.  Unimanual Pen+Touch Input Using Variations of Precision Grip Postures , 2018, UIST.

[17]  Fabrice Matulic,et al.  Sensing techniques for tablet+stylus interaction , 2014, UIST.

[18]  Zhongliang Yang,et al.  Surface EMG-based Sketching Recognition Using Two Analysis Windows and Gene Expression Programming , 2016, Front. Neurosci..

[19]  Yang Li,et al.  Experimental analysis of mode switching techniques in pen-based user interfaces , 2005, CHI.

[20]  Fabrice Matulic,et al.  Supporting active reading on pen and touch-operated tabletops , 2012, AVI.

[21]  Tae-Seong Kim,et al.  3-D hand motion tracking and gesture recognition using a data glove , 2009, 2009 IEEE International Symposium on Industrial Electronics.

[22]  Mike Fraser,et al.  EchoFlex: Hand Gesture Recognition using Ultrasound Imaging , 2017, CHI.

[23]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[24]  Patrick Olivier,et al.  Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor , 2012, UIST.

[25]  Xiang 'Anthony' Chen,et al.  Motion and context sensing techniques for pen computing , 2013, Graphics Interface.

[26]  T. Chau,et al.  Variability of Grip Kinetics during Adult Signature Writing , 2013, PloS one.

[27]  Daniel J. Wigdor,et al.  Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces , 2008, AVI '08.

[28]  Chris Harrison,et al.  Interferi: Gesture Sensing using On-Body Acoustic Interferometry , 2019, CHI.

[29]  Anind K. Dey,et al.  Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch , 2016, CHI.

[30]  Li Cheng,et al.  Efficient Hand Pose Estimation from a Single Depth Image , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  Mikhail A. Lebedev,et al.  Recognition of Handwriting from Electromyography , 2009, PloS one.

[32]  Antonis A. Argyros,et al.  Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[33]  Robert C. Zeleznik,et al.  Hands-on math: a page-based multi-touch and pen desktop for technical work and problem solving , 2010, UIST.

[34]  Yang Zhang,et al.  Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition , 2015, UIST.

[35]  Xiang Cao,et al.  Grips and gestures on a multi-touch pen , 2011, CHI.

[36]  Sungjae Hwang,et al.  MagPen: magnetically driven pen interactions on and around conventional smartphones , 2013, MobileHCI '13.

[37]  Desney S. Tan,et al.  Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces , 2008, CHI.

[38]  Jon Trinder,et al.  The Humane Interface: New Directions for Designing Interactive Systems , 2002, Interact. Learn. Environ..

[39]  Theophanis Tsandilas Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation , 2018, TCHI.

[40]  Jef Raskin,et al.  The Humane Interface: New Directions for Designing Interactive Systems , 2000 .

[41]  Thomas Brox,et al.  Learning to Estimate 3D Hand Pose from Single RGB Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[42]  William Buxton,et al.  Pen + touch = new tools , 2010, UIST.

[43]  Kongqiao Wang,et al.  A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[44]  Patrick Baudisch,et al.  Hover widgets: using the tracking state to extend the capabilities of pen-operated devices , 2006, CHI.

[45]  Fabrice Matulic ColourAIze: AI-Driven Colourisation of Paper Drawings with Interactive Projection System , 2018, ISS.

[46]  Luc Van Gool,et al.  Tracking a hand manipulating an object , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[47]  William Buxton,et al.  Thumb + Pen Interaction on Tablets , 2017, CHI.

[48]  Mathias Wilhelm,et al.  Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction , 2013, AH.

[49]  Antti Oulasvirta,et al.  Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input , 2016, ECCV.

[50]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.