Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry

This paper investigates an emerging input method enabled by progress in hand tracking: input by free motion of fingers. The method is expressive, potentially fast, and usable across many settings as it does not insist on physical contact or visual feedback. Our goal is to inform the design of high-performance input methods by providing detailed analysis of the performance and anatomical characteristics of finger motion. We conducted an experiment using a commercially available sensor to report on the speed, accuracy, individuation, movement ranges, and individual differences of each finger. Findings show differences of up to 50% in movement times and provide indices quantifying the individuation of single fingers. We apply our findings to text entry by computational optimization of multi-finger gestures in mid-air. To this end, we define a novel objective function that considers performance, anatomical factors, and learnability. First investigations of one optimization case show entry rates of 22 words per minute (WPM). We conclude with a critical discussion of the limitations posed by human factors and performance characteristics of existing markerless hand trackers.

[1]  Mark D. Dunlop,et al.  Multidimensional pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking , 2012, CHI.

[2]  L. V. Wassenhove,et al.  A survey of algorithms for the generalized assignment problem , 1992 .

[3]  Michael Rohs,et al.  ShoeSense: a new perspective on gestural interaction and wearable applications , 2012, CHI.

[4]  Shumin Zhai,et al.  SHARK2: a large vocabulary shorthand writing system for pen-based computers , 2004, UIST '04.

[5]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[6]  Douglas C. Engelbart,et al.  A research center for augmenting human intellect , 1968, AFIPS Fall Joint Computing Conference.

[7]  Sterling Orsten,et al.  Dynamics based 3D skeletal hand tracking , 2013, I3D '13.

[8]  Kellogg S. Booth,et al.  Mid-air text input techniques for very large wall displays , 2009, Graphics Interface.

[9]  Tanja Schultz,et al.  Vision-based handwriting recognition for unrestricted text input in mid-air , 2012, ICMI '12.

[10]  Kasper Hornbæk,et al.  Vulture: a mid-air word-gesture keyboard , 2014, CHI.

[11]  Abhishek Ranjan,et al.  Interacting with large displays from a distance with vision-tracked multi-finger gestural input , 2005, SIGGRAPH '06.

[12]  Chris North,et al.  AirStroke: bringing unistroke text entry to freehand gesture interfaces , 2011, CHI.

[13]  Zhi-Hong Mao,et al.  Information Capacity of the Thumb and the Index Finger in Communication , 2009, IEEE Transactions on Biomedical Engineering.

[14]  M. Schieber.  Individuated finger movements of rhesus monkeys: a means of quantifying the independence of the digits. , 1991, Journal of neurophysiology.

[15]  Singiresu S. Rao.  Engineering Optimization : Theory and Practice , 2010 .

[16]  Carlo Tomasi,et al.  Fingerspelling Recognition through Classification of Letter-to-Letter Transitions , 2009, ACCV.

[17]  I. Scott MacKenzie,et al.  Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts' law research in HCI , 2004, Int. J. Hum. Comput. Stud..

[18]  A. Gentile,et al.  Joint control strategies and hand trajectories in multijoint pointing movements. , 1986, Journal of motor behavior.

[19]  Ted Selker,et al.  Multi-finger chords for hand-held tablets: recognizable and memorable , 2014, CHI.

[20]  Bongshin Lee,et al.  Reducing legacy bias in gesture elicitation studies , 2014, INTR.

[21]  Stephen A. Brewster,et al.  Usable gestures for mobile interfaces: evaluating social acceptability , 2010, CHI.

[22]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[23]  Jörg Müller,et al.  Design and evaluation of finger-count interaction: Combining multitouch gestures and menus , 2012, Int. J. Hum. Comput. Stud..

[24]  Per Ola Kristensson,et al.  Continuous recognition of one-handed and two-handed gestures using 3D full-body motion tracking sensors , 2012, IUI '12.

[25]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[26]  G. V. Kondraske,et al.  An angular motion Fitt's Law for human performance modeling and prediction , 1994, Proceedings of 16th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  M H Schieber,et al.  Quantifying the Independence of Human Finger Movements: Comparisons of Digits, Hands, and Movement Frequencies , 2000, The Journal of Neuroscience.

[28]  Per Ola Kristensson,et al.  A versatile dataset for text entry evaluations based on genuine mobile emails , 2011, Mobile HCI.

[29]  Kent Lyons,et al.  Experimental Evaluations of the Twiddler One-Handed Chording Mobile Keyboard , 2006, Hum. Comput. Interact..

[30]  Stephen A. Brewster,et al.  Wrist rotation for interaction in mobile contexts , 2008, Mobile HCI.

[31]  I.,et al.  Fitts' Law as a Research and Design Tool in Human-Computer Interaction , 1992, Hum. Comput. Interact..

[32]  Shumin Zhai,et al.  The metropolis keyboard - an exploration of quantitative techniques for virtual keyboard design , 2000, UIST '00.

[33]  Anders Markussen,et al.  Selection-Based Mid-Air Text Entry on Large Displays , 2013, INTERACT.

[34]  Patrick Olivier,et al.  Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor , 2012, UIST.

[35]  R SEIBEL.  Data Entry through Chord, Parallel Entry Devices , 1964, Human factors.

[36]  D. Chaffin,et al.  An investigation of fitts' law using a wide range of movement amplitudes. , 1976, Journal of motor behavior.

[37]  Antti Oulasvirta,et al.  Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data , 2013, 2013 IEEE International Conference on Computer Vision.

[38]  Chen Qian,et al.  Realtime and Robust Hand Tracking from Depth , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Paul Lukowicz,et al.  Using multiple sensors for mobile sign language recognition , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[40]  Andrew D. Wilson.  Robust computer vision-based detection of pinching for one and two-handed gesture input , 2006, UIST.

[41]  Stephen A. Brewster,et al.  Investigating the effectiveness of tactile feedback for mobile touchscreens , 2008, CHI.

[42]  I. Scott MacKenzie,et al.  Performance differences in the fingers, wrist, and forearm in computer input control , 1997, CHI.

[43]  Shumin Zhai,et al.  Multilingual Touchscreen Keyboard Design and Optimization , 2012, Hum. Comput. Interact..

[44]  Tanja Schultz,et al.  Airwriting: Hands-Free Mobile Text Input by Spotting and Continuous Recognition of 3d-Space Handwriting with Inertial Sensors , 2012, 2012 16th International Symposium on Wearable Computers.

[45]  Sriram Subramanian,et al.  Tilt techniques: investigating the dexterity of wrist-based input , 2009, CHI.

[46]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[47]  M. Turvey,et al.  Intermediate motor learning as decreasing active (dynamical) degrees of freedom , 1998 .