Portal-ble: Intuitive Free-hand Manipulation in Unbounded Smartphone-based Augmented Reality

Smartphone augmented reality (AR) lets users interact with physical and virtual spaces simultaneously. With 3D hand tracking, smartphones become apparatus to grab and move virtual objects directly. Based on design considerations for interaction, mobility, and object appearance and physics, we implemented a prototype for portable 3D hand tracking using a smartphone, a Leap Motion controller, and a computation unit. Following an experience prototyping procedure, 12 researchers used the prototype to help explore usability issues and define the design space. We identified issues in perception (moving to the object, reaching for the object), manipulation (successfully grabbing and orienting the object), and behavioral understanding (knowing how to use the smartphone as a viewport). To overcome these issues, we designed object-based feedback and accommodation mechanisms and studied their perceptual and behavioral effects via two tasks: picking up distant objects, and assembling a virtual house from blocks. Our mechanisms enabled significantly faster and more successful user interaction than the initial prototype in picking up and manipulating stationary and moving objects, with a lower cognitive load and greater user preference. The resulting system---Portal-ble---improves user intuition and aids free-hand interactions in mobile situations.

[1]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[2]  Weidong Huang,et al.  Recent Trends of Mobile Collaborative Augmented Reality Systems , 2014 .

[3]  Mark Billinghurst,et al.  Spatial sound localization in an augmented reality environment , 2006, OZCHI.

[4]  Keith Cheverst,et al.  Developing a context-aware electronic tourist guide: some issues and experiences , 2000, CHI.

[5]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[6]  Hirokazu Kato,et al.  The effects of shadow representation of virtual objects in augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[7]  Tobias Höllerer,et al.  Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[8]  Antti Oulasvirta,et al.  Performance and Ergonomics of Touch Surfaces: A Comparative Study using Biomechanical Simulation , 2015, CHI.

[9]  Paul Coulton,et al.  3D Virtual Tracing and Depth Perception Problem on Mobile AR , 2016, CHI Extended Abstracts.

[10]  Jong-Il Park,et al.  One-handed interaction with augmented virtual objects on mobile devices , 2008, VRCAI.

[11]  Hannes Kaufmann,et al.  3DTouch and HOMER-S: intuitive manipulation techniques for one-handed handheld augmented reality , 2013, VRIC.

[12]  David A. Forsyth,et al.  BeThere: 3D mobile collaboration with spatial input , 2013, CHI.

[13]  Chen Qian,et al.  Realtime and Robust Hand Tracking from Depth , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Steven K. Feiner,et al.  Perceptual issues in augmented reality revisited , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[15]  Bruce H. Thomas,et al.  A wearable computer system with augmented reality to support terrestrial navigation , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[16]  Ke Wang,et al.  AI Benchmark: Running Deep Neural Networks on Android Smartphones , 2018, ECCV Workshops.

[17]  Weidong Huang,et al.  HandsOnVideo: Towards a Gesture based Mobile AR System for Remote Collaboration , 2011 .

[18]  R. George,et al.  Human finger types , 1930 .

[19]  David Kim,et al.  HoloDesk: direct 3d interactions with a situated see-through display , 2012, CHI.

[20]  Zhenyi He,et al.  Hand-based interaction for object manipulation with augmented reality glasses , 2014, VRCAI '14.

[21]  Danica Kragic,et al.  Monocular real-time 3D articulated hand pose estimation , 2009, 2009 9th IEEE-RAS International Conference on Humanoid Robots.

[22]  Shahrouz Yousefi,et al.  3D gesture-based interaction for immersive experience in mobile VR , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[23]  Tom Drummond,et al.  Going out: robust model-based tracking for outdoor augmented reality , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[24]  Vincent Lepetit,et al.  Hands Deep in Deep Learning for Hand Pose Estimation , 2015, ArXiv.

[25]  Bruce H. Thomas,et al.  ARQuake: an outdoor/indoor augmented reality first person application , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[26]  Ramakrishnan Mukundan,et al.  3D gesture interaction for handheld augmented reality , 2014, SIGGRAPH ASIA Mobile Graphics and Interactive Applications.

[27]  Wolfgang Birkfellner,et al.  A head-mounted operating binocular for augmented reality visualization in medicine - design and initial evaluation , 2002, IEEE Transactions on Medical Imaging.

[28]  Jae Yeol Lee,et al.  Direct hand touchable interactions in augmented reality environments for natural and intuitive user experiences , 2013, Expert Syst. Appl..

[29]  Richard D. Green,et al.  An advanced interaction framework for augmented reality based exposure treatment , 2013, 2013 IEEE Virtual Reality (VR).

[30]  Pourang Irani,et al.  Consumed endurance: a metric to quantify arm fatigue of mid-air interactions , 2014, CHI.

[31]  Pan Hui,et al.  A Survey on Haptic Technologies for Mobile Augmented Reality , 2017, ACM Comput. Surv..

[32]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[33]  Jens Grubert,et al.  Designing mobile augmented reality , 2013, MobileHCI '13.

[34]  Scott D. Roth,et al.  Ray casting for modeling solids , 1982, Comput. Graph. Image Process..

[35]  Wolfgang Hürst,et al.  Gesture-based interaction via finger tracking for mobile augmented reality , 2011, Multimedia Tools and Applications.

[36]  Minseok Kim,et al.  Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality , 2016, Multimedia Tools and Applications.

[37]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[38]  Danielle Albers Szafir,et al.  Interacting with Distant Objects in Augmented Reality , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[39]  Varun Jain,et al.  GestARLite: An On-Device Pointing Finger Based Gestural Interface for Smartphones and Video See-Through Head-Mounts , 2019, ArXiv.

[40]  Christian Theobalt,et al.  Real-Time Hand Tracking Under Occlusion from an Egocentric RGB-D Sensor , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[41]  Otmar Hilliges,et al.  In-air gestures around unmodified mobile devices , 2014, UIST.

[42]  Mark Billinghurst,et al.  Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[43]  Jane Fulton Suri,et al.  Experience prototyping , 2000, DIS '00.

[44]  Hyotaek Lim,et al.  Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware , 2015, Multimedia Tools and Applications.

[45]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[46]  Phivos Mylonas,et al.  Natural Interaction in Augmented Reality Context , 2019, VIPERC@IRCDL.

[47]  Brendan T Barrett,et al.  Changes to control of adaptive gait in individuals with long-standing reduced stereoacuity. , 2010, Investigative ophthalmology & visual science.

[48]  Christian Theobalt,et al.  GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  Rod McCall,et al.  TimeWarp: interactive time travel with a mobile mixed reality game , 2008, Mobile HCI.

[50]  Jihad El-Sana,et al.  Free-hand interaction for handheld augmented reality using an RGB-depth camera , 2013, SA '13.

[51]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.