Gesture-based interaction via finger tracking for mobile augmented reality

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.

[1]  David A. Atchison,et al.  Chapter 20 – The aging eye , 2000 .

[2]  Pattie Maes,et al.  SixthSense: a wearable gestural interface , 2009, SIGGRAPH ASIA Art Gallery & Emerging Technologies.

[3]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[4]  David A. Atchison,et al.  Optics of the Human Eye , 2023 .

[5]  Paul Coulton,et al.  Motion controlled mobile 3D multiplayer gaming , 2008, ACE '08.

[6]  Dieter Schmalstieg,et al.  Finger tracking for interaction in augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[7]  Kenneth J. Waldron,et al.  The Workspaces of a Mechanical Manipulator , 1981 .

[8]  Mark Ollila,et al.  Interactive Collaborative Scene Assembly Using AR on Mobile Phones , 2006, ICAT.

[9]  María Menéndez,et al.  Behand: augmented virtuality gestural interaction for mobile phones , 2010, Mobile HCI.

[10]  Yang Li,et al.  DoubleFlip: a motion gesture delimiter for mobile interaction , 2010, UIST '10.

[11]  Stephen A. Brewster,et al.  Usable gestures for mobile interfaces: evaluating social acceptability , 2010, CHI.

[12]  Jihad El-Sana,et al.  In-place 3D sketching for authoring and augmenting mechanical systems , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[13]  Michael Rohs,et al.  Visual code widgets for marker-based interaction , 2005, 25th IEEE International Conference on Distributed Computing Systems Workshops.

[14]  Dieter Schmalstieg,et al.  Experiences with Handheld Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[15]  Dieter Schmalstieg,et al.  Making Augmented Reality Practical on Mobile Phones, Part 1 , 2009, IEEE Computer Graphics and Applications.

[16]  Michael Rohs,et al.  Real-World Interaction with Camera Phones , 2004, UCS.

[17]  James L. Crowley,et al.  Finger Tracking as an Input Device for Augmented Reality , 1995 .

[18]  Mark Billinghurst,et al.  Virtual object manipulation using a mobile phone , 2005, ICAT '05.

[19]  Sukeshini A. Grandhi,et al.  Understanding naturalness and intuitiveness in gesture production: insights for touchless gestural interfaces , 2011, CHI.

[20]  R. Green,et al.  3D natural hand interaction for AR applications , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[21]  Haibo Li,et al.  3D Gestural Interaction for Stereoscopic Visualization on Mobile Devices , 2011, CAIP.

[22]  Hirokazu Kato,et al.  Advanced Interaction Techniques for Augmented Reality Applications , 2009, HCI.

[23]  Blair MacIntyre,et al.  Art of defense: a collaborative handheld augmented reality board game , 2009, Sandbox@SIGGRAPH.

[24]  M. Weiser,et al.  An empirical comparison of pie vs. linear menus , 1988, CHI '88.

[25]  Dieter Schmalstieg,et al.  Robust and unobtrusive marker tracking on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[26]  Hyun Seung Yang,et al.  Real-time robust body part tracking for augmented reality interface , 2009, VRCAI '09.

[27]  Woontack Woo,et al.  An empirical evaluation of virtual hand techniques for 3D object manipulation in a tangible augmented reality environment , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[28]  Ernesto Damiani,et al.  Augmented reality technologies, systems and applications , 2010, Multimedia Tools and Applications.

[29]  Dieter Schmalstieg,et al.  Multiple target detection and tracking with guaranteed framerates on mobile phones , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[30]  Jong-Il Park,et al.  One-handed interaction with augmented virtual objects on mobile devices , 2008, VRCAI.

[31]  Wolfgang Hürst,et al.  Multimodal Interaction Concepts for Mobile Augmented Reality Applications , 2011, MMM.

[32]  Michael Rohs,et al.  Interaction with magic lenses: real-world validation of a Fitts' Law model , 2011, CHI.

[33]  Mark Billinghurst,et al.  Using a mobile phone for 6 DOF mesh editing , 2007, CHINZ.

[34]  P. Coulton,et al.  3D Motion Control of Connected Augmented Virtuality on Mobile Phones , 2008, 2008 International Symposium on Ubiquitous Virtual Reality.

[35]  Masatoshi Ishikawa,et al.  In-air typing interface for mobile devices with vibration feedback , 2010, SIGGRAPH '10.

[36]  Yang Li,et al.  DoubleFlip: a motion gesture delimiter for interaction , 2010, UIST '10.

[37]  Jihad El-Sana,et al.  Shape Recognition and Pose Estimation for Mobile Augmented Reality , 2009, IEEE Transactions on Visualization and Computer Graphics.

[38]  Peter Fröhlich,et al.  Markerless visual fingertip detection for natural mobile device interaction , 2011, Mobile HCI.

[39]  Dieter Schmalstieg,et al.  Making Augmented Reality Practical on Mobile Phones, Part 2 , 2009, IEEE Computer Graphics and Applications.

[40]  Sylvain Paris,et al.  6D hands: markerless hand-tracking for computer aided design , 2011, UIST.

[41]  Pattie Maes,et al.  WUW - wear Ur world: a wearable gestural interface , 2009, CHI Extended Abstracts.

[42]  Yoichi Sato,et al.  Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system , 2001, TCHI.

[43]  Masatoshi Ishikawa,et al.  Fast finger tracking system for in-air typing interface , 2009, CHI Extended Abstracts.

[44]  Darren Cosker,et al.  3D Gesture Recognition: An Evaluation of User and System Performance , 2011, Pervasive.

[45]  Mark Billinghurst,et al.  Mobile phone based AR scene assembly , 2005, MUM '05.

[46]  Paul Coulton,et al.  Using “tilt” as an interface to control “no-button” 3-D mobile games , 2008, CIE.