3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review

Handheld mobile devices referring to mobile phones and tablets become the major output medium for augmented reality (AR), which have seen significant growth in popularity and usage among the public due to the growing release of consumer-oriented communication products nowadays, especially touchscreen smartphones. Unlike traditional desktop or tabletop AR (large display-based) and head-mounted display-based AR systems, small display-based AR (handheld mobile display) requires different interaction techniques that mostly utilize single-hand interaction as well as the limitation of small screen display and limited activity time due to the battery operation hour of handheld mobile devices. However, in handheld mobile AR, research is still lacking, especially research that focuses on 3D interaction in handheld mobile AR for virtual object manipulation. Thus, this paper provides an overview of 3D interaction techniques in handheld mobile AR with critical analysis. First, we describe three main interaction technique categories that applicable in handheld mobile AR, which is touch-based interaction, mid-air gestures-based interaction, and device-based interaction techniques, of their basic concepts on 3D object manipulation. Then, we classify and systematize the highlighted techniques and discuss the advantages and drawbacks of each. Previous studies for widely used techniques have been studied comprehensively. We then draw up a comparison among the different techniques based on the important elements considered in handheld mobile AR. The aim of this paper is to provide researchers with background information on AR and those who are interested in the field of 3D interaction, realizing each technique category.

[1]  Ben Shneiderman,et al.  High Precision Touchscreens: Design Strategies and Comparisons with a Mouse , 1991, Int. J. Man Mach. Stud..

[2]  Mark Billinghurst,et al.  Virtual object manipulation using a mobile phone , 2005, ICAT '05.

[3]  Jihad El-Sana,et al.  Free-hand interaction for handheld augmented reality using an RGB-depth camera , 2013, SA '13.

[4]  Martin Hachet,et al.  tBox: a 3d transformation widget designed for touch-screens , 2011, CHI.

[5]  Kaisa Väänänen,et al.  Expected user experience of mobile augmented reality services: a user study in the context of shopping centres , 2011, Personal and Ubiquitous Computing.

[6]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[7]  Steven K. Feiner,et al.  Mobile augmented reality interaction techniques for authoring situated media on-site , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[8]  Laurent Grisoni,et al.  Integrality and Separability of Multitouch Interaction Techniques in 3D Manipulation Tasks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[9]  Jun Rekimoto,et al.  Tilting operations for small screen interfaces , 1996, UIST '96.

[10]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[11]  Patrick Baudisch,et al.  Back-of-device interaction allows creating very small touch devices , 2009, CHI.

[12]  James H. Oliver,et al.  Augmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty , 2015, Int. J. Hum. Comput. Interact..

[13]  Jun Rekimoto,et al.  SmartSkin: an infrastructure for freehand manipulation on interactive surfaces , 2002, CHI.

[14]  J. Kumar,et al.  Recent Development of Augmented Reality in Surgery: A Review , 2017, Journal of healthcare engineering.

[15]  M. Sheelagh T. Carpendale,et al.  Rotation and translation mechanisms for tabletop interaction , 2006, First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06).

[16]  Anatole Lécuyer,et al.  Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback , 2009, PRESENCE: Teleoperators and Virtual Environments.

[17]  Mark Billinghurst,et al.  Mobile phone based AR scene assembly , 2005, MUM '05.

[18]  Jakub Segen,et al.  Gesture VR: vision-based 3D hand interace for spatial interaction , 1998, MULTIMEDIA '98.

[19]  T. Langlotz AR 2.0: Social Media in Mobile Augmented Reality , 2013 .

[20]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[21]  Jovan Popović,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH 2009.

[22]  Gino Dini,et al.  Application of Augmented Reality Techniques in Through-life Engineering Services , 2015 .

[23]  Laurent Grisoni,et al.  The design and evaluation of 3D positioning techniques for multi-touch displays , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[24]  Daniel Mendes,et al.  Mid-air interactions above stereoscopic interactive tables , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[25]  Peng Du,et al.  A Novel Multi-touch Approach for 3D Object Free Manipulation , 2017, AniNex.

[26]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[27]  S. Kurkovsky,et al.  Current issues in handheld augmented reality , 2012, 2012 International Conference on Communications and Information Technology (ICCIT).

[28]  Tiago M. Fernández-Caramés,et al.  A Review on Industrial Augmented Reality Systems for the Industry 4.0 Shipyard , 2018, IEEE Access.

[29]  Ana Paula Ambrosio,et al.  Use of Augmented Reality to Support Education - Creating a Mobile E-learning Tool and using it with an Inquiry-based Approach , 2017, CSEDU.

[30]  Klaus H. Hinrichs,et al.  Triangle cursor: interactions with objects above the tabletop , 2011, ITS '11.

[31]  Roderick Murray-Smith,et al.  Dynamics of tilt-based browsing on mobile devices , 2007, CHI Extended Abstracts.

[32]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[33]  Hirokazu Kato,et al.  SlidAR: A 3D positioning method for SLAM-based handheld augmented reality , 2016, Comput. Graph..

[34]  Philip L. Davidson,et al.  A screen-space formulation for 2D and 3D direct manipulation , 2009, UIST '09.

[35]  Peter Fröhlich,et al.  Markerless visual fingertip detection for natural mobile device interaction , 2011, Mobile HCI.

[36]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[37]  S. J. Steiner,et al.  Virtual reality and augmented reality as a training tool for assembly tasks , 1999, 1999 IEEE International Conference on Information Visualization (Cat. No. PR00210).

[38]  Daniel Vogel,et al.  Shift: a technique for operating pen-based interfaces using touch , 2007, CHI.

[39]  Jens Grubert,et al.  Augmented Reality Browser Survey , 2012 .

[40]  Saul Greenberg,et al.  The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface , 2011, INTERACT.

[41]  François Bérard,et al.  Two-finger 3D rotations for novice users: surjective and integral interactions , 2014, AVI.

[42]  Domenico Prattichizzo,et al.  Using Kinect for hand tracking and rendering in wearable haptics , 2011, 2011 IEEE World Haptics Conference.

[43]  Dieter Schmalstieg,et al.  Finger tracking for interaction in augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[44]  Steven K. Feiner,et al.  Balloon Selection: A Multi-Finger Technique for Accurate Low-Fatigue 3D Selection , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[45]  Otmar Hilliges,et al.  Bringing physics to the surface , 2008, UIST '08.

[46]  Alexander Zelinsky,et al.  Visual gesture interfaces for virtual environments , 2002, Interact. Comput..

[47]  Grenoble Cede,et al.  Visual Tracking of Bare Fingers for Interactive Surfaces , 2004 .

[48]  Andrew S. Forsberg,et al.  Image plane interaction techniques in 3D immersive environments , 1997, SI3D.

[49]  J. Shuhaiber,et al.  Augmented reality in surgery. , 2004, Archives of surgery.

[50]  Andreas Butz,et al.  Interactions in the air: adding further depth to interactive tabletops , 2009, UIST '09.

[51]  Mark Billinghurst,et al.  Freeze view touch and finger gesture based interaction methods for handheld augmented reality interfaces , 2012, IVCNZ '12.

[52]  Takashi Komuro,et al.  [Paper] 3D Interaction with Virtual Objects in a Precisely-aligned View using a See-through Mobile AR System , 2017 .

[53]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[54]  Joaquim A. Jorge,et al.  Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space , 2012, Graphics Interface.

[55]  Joseph J. LaViola,et al.  Poster: Real-time markerless kinect based finger tracking and hand gesture recognition for HCI , 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI).

[56]  Timo Engelke,et al.  Augmented Reality Training for Assembly and Maintenance Skills , 2011 .

[57]  Tsukasa Ogasawara,et al.  Haptic augmented reality interface using the real force response of an object , 2009, VRST '09.

[58]  Joe Marshall,et al.  Experiments in 3D interaction for mobile phone AR , 2007, GRAPHITE '07.

[59]  Pourang Irani,et al.  Consumed endurance: a metric to quantify arm fatigue of mid-air interactions , 2014, CHI.

[60]  Wolfgang Stuerzlinger,et al.  A one-handed multi-touch mating method for 3d rotations , 2013, CHI Extended Abstracts.

[61]  Lei Gao,et al.  Poster: Markerless fingertip-based 3D interaction for handheld augmented reality in a small workspace , 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI).

[62]  Ehud Sharlin,et al.  Napkin sketch: handheld mixed reality 3D sketching , 2008, VRST '08.

[63]  Jihad El-Sana,et al.  Markerless 3D gesture-based interaction for handheld Augmented Reality interfaces , 2013, ISMAR.

[64]  Ajune Wanis Ismail,et al.  Designing an Augmented Reality Multimodal Interface for 6DOF Manipulation Techniques - Multimodal Fusion Using Gesture and Speech Input for AR , 2018, IntelliSys.

[65]  Hirokazu Kato,et al.  Handheld Guides in Inspection Tasks: Augmented Reality versus Picture , 2018, IEEE Transactions on Visualization and Computer Graphics.

[66]  Rafael Radkowski,et al.  Interactive Hand Gesture-based Assembly for Augmented Reality Applications , 2012, ACHI 2012.

[67]  Mehdi Mekni,et al.  Augmented Reality : Applications , Challenges and Future Trends , 2014 .

[68]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[69]  C. Jennings,et al.  Robust finger tracking with multiple cameras , 1999, Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. In Conjunction with ICCV'99 (Cat. No.PR00378).

[70]  Peng Song,et al.  A handle bar metaphor for virtual object manipulation with mid-air interaction , 2012, CHI.

[71]  Daniel Mendes,et al.  Using custom transformation axes for mid-air manipulation of 3D virtual objects , 2017, VRST.

[72]  Wolfgang Hürst,et al.  Gesture-based interaction via finger tracking for mobile augmented reality , 2011, Multimedia Tools and Applications.

[73]  Minseok Kim,et al.  Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality , 2016, Multimedia Tools and Applications.

[74]  Pan Hui,et al.  Mobile Augmented Reality Survey: From Where We Are to Where We Go , 2017, IEEE Access.

[75]  Weiqin Chen,et al.  Historical Oslo on a Handheld Device - A Mobile Augmented Reality Application , 2014, KES.

[76]  Woontack Woo,et al.  3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint , 2015, IEEE Transactions on Visualization and Computer Graphics.

[77]  Steven K. Feiner,et al.  Perceptual issues in augmented reality revisited , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[78]  Lena Maier-Hein,et al.  Mobile markerless augmented reality and its application in forensic medicine , 2015, International Journal of Computer Assisted Radiology and Surgery.

[79]  Hannes Kaufmann,et al.  Evaluating RGB+D hand posture detection methods for mobile 3D interaction , 2014, VRIC.

[80]  Karljohan E. Lundin Palmerius,et al.  A study on improving close and distant device movement pose manipulation for hand-held augmented reality , 2016, VRST.

[81]  Mohd Shahrizal Sunar,et al.  A REVIEW OF 3D GESTURE INTERACTION FOR HANDHELD AUGMENTED REALITY , 2015 .

[82]  Hirokazu Kato,et al.  Advanced Interaction Techniques for Augmented Reality Applications , 2009, HCI.

[83]  Blair MacIntyre,et al.  Art of defense: a collaborative handheld augmented reality board game , 2009, Sandbox@SIGGRAPH.

[84]  D. Mourtzis,et al.  Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry , 2017 .

[85]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[86]  Jong-Il Park,et al.  One-handed interaction with augmented virtual objects on mobile devices , 2008, VRCAI.

[87]  Hannes Kaufmann,et al.  3DTouch and HOMER-S: intuitive manipulation techniques for one-handed handheld augmented reality , 2013, VRIC.

[88]  Tolga K. Çapin,et al.  Dual-Finger 3D Interaction Techniques for mobile devices , 2012, Personal and Ubiquitous Computing.

[89]  Tobias Höllerer,et al.  Real-time hand interaction for augmented reality on mobile phones , 2013, IUI '13.

[90]  Lester Madden Professional Augmented Reality Browsers for Smartphones: Programming for junaio, Layar and Wikitude , 2011 .

[91]  Jin Sung Choi,et al.  Freeze-Set-Go interaction method for handheld mobile augmented reality environments , 2009, VRST '09.

[92]  Hyun Seung Yang,et al.  Smartphone as an augmented reality authoring tool via multi-touch based 3D interaction method , 2012, VRCAI '12.

[93]  M. Sheelagh T. Carpendale,et al.  Sticky tools: full 6DOF force-based interaction for multi-touch tables , 2009, ITS '09.

[94]  Jun Rekimoto,et al.  A New You: From Augmented Reality to Augmented Human , 2014, ITS '14.

[95]  Arindam Dey,et al.  A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014 , 2018, Front. Robot. AI.

[96]  Woontack Woo,et al.  An empirical evaluation of virtual hand techniques for 3D object manipulation in a tangible augmented reality environment , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[97]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[98]  Takuji Narumi,et al.  Integrated view-input ar interaction for virtual object manipulation using tablets and smartphones , 2015, Advances in Computer Entertainment.

[99]  Xiang 'Anthony' Chen,et al.  The fat thumb: using the thumb's contact size for single-handed mobile interaction , 2012, Mobile HCI.

[100]  Shion Yeing Ooi Mobile augmented reality for smart pet , 2019 .

[101]  Ramakrishnan Mukundan,et al.  3D gesture interaction for handheld augmented reality , 2014, SIGGRAPH ASIA Mobile Graphics and Interactive Applications.

[102]  Michael Rohs,et al.  PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices , 2012, AVI.

[103]  Marie-Paule Cani,et al.  Understanding Hand Degrees of Freedom and Natural Gestures for 3D Interaction on Tabletop , 2013, INTERACT.

[104]  Mark Billinghurst,et al.  Face to face collaborative AR on mobile phones , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[105]  Martin Hachet,et al.  Combining multi-touch input and device movement for 3D manipulations in mobile augmented reality environments , 2014, SUI.

[106]  Hongbo Fu,et al.  Two‐Finger Gestures for 6DOF Manipulation of 3D Objects , 2012, Comput. Graph. Forum.

[107]  Yanqing Wang,et al.  The structure of object transportation and orientation in human-computer interaction , 1998, CHI.

[108]  Blair MacIntyre,et al.  BragFish: exploring physical and social interaction in co-located handheld augmented reality games , 2008, ACE '08.

[109]  Takeo Kanade,et al.  DigitEyes: vision-based hand tracking for human-computer interaction , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[110]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .