Pre-Touch Sensing for Mobile Interaction

Touchscreens continue to advance including progress towards sensing fingers proximal to the display. We explore this emerging pre-touch modality via a self-capacitance touchscreen that can sense multiple fingers above a mobile device, as well as grip around the screen's edges. This capability opens up many possibilities for mobile interaction. For example, using pre-touch in an anticipatory role affords an "ad-lib interface" that fades in a different UI--appropriate to the context--as the user approaches one-handed with a thumb, two-handed with an index finger, or even with a pinch or two thumbs. Or we can interpret pre-touch in a retroactive manner that leverages the approach trajectory to discern whether the user made contact with a ballistic vs. a finely-targeted motion. Pre-touch also enables hybrid touch + hover gestures, such as selecting an icon with the thumb while bringing a second finger into range to invoke a context menu at a convenient location. Collectively these techniques illustrate how pre-touch sensing offers an intriguing new back-channel for mobile interaction.

[1]  M. Jeannerod,et al.  Constraints on human arm movement trajectories. , 1987, Canadian journal of psychology.

[2]  Xiang Cao,et al.  Grips and gestures on a multi-touch pen , 2011, CHI.

[3]  Guy Weinzapfel,et al.  One-point touch input of vector information for computer displays , 1978, SIGGRAPH '78.

[4]  Shwetak N. Patel,et al.  ContextType: using hand posture information to improve mobile touch screen text entry , 2013, CHI.

[5]  François Guimbretière,et al.  FlexAura: a flexible near-surface range sensor , 2012, UIST '12.

[6]  Otmar Hilliges,et al.  In-air gestures around unmodified mobile devices , 2014, UIST.

[7]  Daniel F. Keefe,et al.  Nailing down multi-touch: anchored above the surface interaction for 3D modeling and navigation , 2012, Graphics Interface.

[8]  Francis K. H. Quek,et al.  Action-Transferred Navigation Technique Design Approach Supporting Human Spatial Learning , 2015, TCHI.

[9]  Joanna McGrenere,et al.  Grip Change as an Information Side Channel for Mobile Touch Interaction , 2015, CHI.

[10]  Wei Sun,et al.  The role of printing parameters and scaffold biopolymer properties in the efficacy of a new hybrid nano-bioprinting system , 2009, Biofabrication.

[11]  Joshua R. Smith,et al.  Electric field imaging pretouch for robotic graspers , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.

[13]  Steven Bathiche,et al.  A practical pressure sensitive computer keyboard , 2009, UIST '09.

[14]  Michael Rohs,et al.  A Taxonomy of Microinteractions: Defining Microgestures Based on Ergonomic and Scenario-Dependent Requirements , 2011, INTERACT.

[15]  Kee-Eung Kim,et al.  Hand Grip Pattern Recognition for Mobile User Interfaces , 2006, AAAI.

[16]  William Buxton,et al.  A three-state model of graphical input , 1990, INTERACT.

[17]  Sebastian Boring,et al.  HandSense: discriminating different ways of grasping and holding a tangible user interface , 2009, Tangible and Embedded Interaction.

[18]  Simon Rogers,et al.  AnglePose: robust, precise capacitive touch tracking via 3d orientation estimation , 2011, CHI.

[19]  Markus Wacker,et al.  Enhanced feed-forward for a user aware multi-touch device , 2012, NordiCHI.

[20]  John C. Tang,et al.  VideoWhiteboard: video shadows to support remote collaboration , 1991, CHI.

[21]  Dimitre Novatchev,et al.  Chunking and Phrasing and the Design of Human-Computer Dialogues - Response , 1986, IFIP Congress.

[22]  Geehyuk Lee,et al.  Force gestures: augmenting touch screen gestures with normal and tangential forces , 2011, UIST.

[23]  Daniel Vogel,et al.  Pin-and-Cross: A Unimanual Multitouch Technique Combining Static Touches with Crossing Selection , 2015, UIST.

[24]  Peter Brandl,et al.  Occlusion-aware menu design for digital tabletops , 2009, CHI Extended Abstracts.

[25]  Tovi Grossman,et al.  A probabilistic approach to modeling two-dimensional pointing , 2005, TCHI.

[26]  Michael Rohs,et al.  HoverFlow: expanding the design space of around-device interaction , 2009, Mobile HCI.

[27]  Christine L. MacKenzie,et al.  The Grasping Hand , 2011, The Grasping Hand.

[28]  Joanna Bergstrom-Lehtovirta,et al.  Modeling the functional area of the thumb on mobile touchscreen surfaces , 2014, CHI.

[29]  B. Bederson,et al.  Understanding Single-Handed Mobile Device Interaction , 2006 .

[30]  Roy Want,et al.  Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces , 1998, CHI.

[31]  Tovi Grossman,et al.  Medusa: a proximity-aware multi-touch tabletop , 2011, UIST.

[32]  Y. Guiard.  Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. , 1987, Journal of motor behavior.

[33]  Anoop Gupta,et al.  Exploring and Understanding Unintended Touch during Direct Pen Interaction , 2014, TCHI.

[34]  Katrin Wolf,et al.  PinchPad: performance of touch-based gestures while grasping devices , 2012, TEI.

[35]  Shwetak N. Patel,et al.  GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones , 2012, UIST.

[36]  V. Michael Bove,et al.  Graspables: grasp-recognition as a user interface , 2009, CHI.

[37]  Xing-Dong Yang,et al.  TouchCuts and TouchZoom: enhanced target selection for touch displays using finger proximity sensing , 2011, CHI.

[38]  Fabrice Matulic,et al.  Sensing techniques for tablet+stylus interaction , 2014, UIST.

[39]  Jose L. Contreras-Vidal,et al.  Understanding One-Handed Use of Mobile Devices , 2008 .

[40]  Ming-Sui Lee,et al.  iRotateGrasp: automatic screen rotation based on grasp of mobile devices , 2013, CHI Extended Abstracts.

[41]  Andreas Butz,et al.  Interactions in the air: adding further depth to interactive tabletops , 2009, UIST '09.

[42]  Yang Li,et al.  Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures , 2014, UIST.

[43]  Yang Li,et al.  Experimental analysis of mode switching techniques in pen-based user interfaces , 2005, CHI.

[44]  Saul Greenberg,et al.  The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface , 2011, INTERACT.

[45]  Simon Rogers,et al.  28 frames later: predicting screen touches from back-of-device grip changes , 2014, CHI.

[46]  Patrick Baudisch,et al.  The generalized perceived input point model and how to double touch accuracy by extracting fingerprints , 2010, CHI.

[47]  Xiang 'Anthony' Chen,et al.  Air+touch: interweaving touch & in-air gestures , 2014, UIST.

[48]  John C. Tang,et al.  Three's company: understanding communication channels in three-way distributed collaboration , 2010, CSCW '10.

[49]  Raimund Dachselt,et al.  Revisiting hovering: interaction guides for interactive surfaces , 2012, ITS.

[50]  Daniel Vogel,et al.  Occlusion-aware interfaces , 2010, CHI.

[51]  Eric Horvitz,et al.  Foreground and background interaction with sensor-enhanced mobile devices , 2005, TCHI.

[52]  Patrick Baudisch,et al.  Hover widgets: using the tracking state to extend the capabilities of pen-operated devices , 2006, CHI.

[53]  Caroline Appert,et al.  Prospective motor control on tabletops: planning grasp for multitouch interaction , 2014, CHI.

[54]  Daniel J. Wigdor,et al.  Zero-latency tapping: using hover information to predict touch locations and eliminate touchdown latency , 2014, UIST.

[55]  Ravin Balakrishnan,et al.  Fitts' law and expanding targets: Experimental studies and designs for user interfaces , 2005, TCHI.

[56]  Pourang Irani,et al.  Sensing Tablet Grasp + Micro-mobility for Active Reading , 2015, UIST.

[57]  Hyunjeong Lee,et al.  New mobile UI with hand-grip recognition , 2009, CHI Extended Abstracts.

[58]  Roderick Murray-Smith,et al.  Focused and casual interactions: allowing users to vary their level of engagement , 2013, CHI.

[59]  Austin Henderson,et al.  Making sense of sensing systems: five questions for designers and researchers , 2002, CHI.

[60]  Jan Stage,et al.  Handbook of Research on User Interface Design and Evaluation for Mobile Technology , 2008 .

[61]  Sarah H. Creem,et al.  Grasping objects by their handles: a necessary interaction between cognition and action. , 2001, Journal of experimental psychology. Human perception and performance.

[62]  Mike Y. Chen,et al.  iGrasp: grasp-based adaptive keyboard for mobile devices , 2013, CHI Extended Abstracts.

[63]  Itiro Siio,et al.  Mobile interaction using paperweight metaphor , 2006, CHI Extended Abstracts.