ARCHIE: A User-Focused Framework for Testing Augmented Reality Applications in the Wild

In this paper, we present ARCHIE, a framework for testing augmented reality applications in the wild. ARCHIE collects user feedback and system state data in situ to help developers identify and debug issues important to testers. It also supports testing of multiple application versions (called "profiles") in a single evaluation session, prioritizing those versions which the tester finds more appealing. To evaluate ARCHIE, we implemented four distinct test case applications and used these applications to examine the performance overhead and context switching cost of incorporating our framework into a pre-existing code base. With these, we demonstrate that ARCHIE provides no significant overhead for AR applications, and introduces at most 2% processing overhead when switching among large groups of testable profiles.

[1]  Pan Hui,et al.  Talk2Me: A Framework for Device-to-Device Augmented Reality Social Network , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[2]  Danielle Albers Szafir,et al.  Designing for Depth Perceptions in Augmented Reality , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[3]  Holger Regenbrecht,et al.  PanoVC: Pervasive telepresence using mobile phones , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[4]  Marina-Anca Cidotã,et al.  Assessing Upper Extremity Motor Dysfunction Using an Augmented Reality Game , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[5]  S. Adamovich,et al.  Virtual reality-augmented rehabilitation for patients following stroke. , 2002, Physical therapy.

[6]  Tzu-Chien Liu,et al.  The Influence of using Augmented Reality on Textbook Support for Learners of Different Learning Styles , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[7]  Jorge Bacca,et al.  Augmented Reality Trends in Education: A Systematic Review of Research and Applications , 2014, J. Educ. Technol. Soc..

[8]  Yuan-Cheng Lai,et al.  Improving the Accuracy of Automated GUI Testing for Embedded Systems , 2014, IEEE Software.

[9]  Mark Billinghurst,et al.  A User Study on MR Remote Collaboration Using Live 360 Video , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[10]  Chris Harrison,et al.  EyeSpyVR: Interactive Eye Sensing Using Off-the-Shelf, Smartphone-Based VR Headsets , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[11]  Jerry Zeyu Gao,et al.  Modeling mobile application test platform and environment: testing criteria and complexity analysis , 2014, JAMAICA 2014.

[12]  Silke Geisen.  Augmented Reality in surgery , 2005 .

[13]  Alessandro Orso,et al.  X-PERT: a web application testing tool for cross-browser inconsistency detection , 2014, ISSTA 2014.

[14]  Kangsoo Kim,et al.  Does a Digital Assistant Need a Body? The Influence of Visual Embodiment and Social Behavior on the Perception of Intelligent Virtual Agents in AR , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[15]  Sangki Yun,et al.  When IoT met Augmented Reality: Visualizing the Source of the Wireless Signal in AR View , 2019, MobiSys.

[16]  Sebastian Pape,et al.  Exploring the Hype: Investigating Technology Acceptance Factors of Pokémon Go , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[17]  Patrick Olivier,et al.  LApp: A Speech Loudness Application for People with Parkinson's on Google Glass , 2015, CHI.

[18]  Antonio Krüger,et al.  VRShop: A Mobile Interactive Virtual Reality Shopping Environment Combining the Benefits of On- and Offline Shopping , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[19]  Holger Regenbrecht,et al.  Augmented reality projects in the automotive and aerospace industries , 2005, IEEE Computer Graphics and Applications.

[20]  Ludger Schmidt,et al.  Text Presentation for Augmented Reality Applications in Dual-Task Situations , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[21]  Marc Erich Latoschik,et al.  Immersive Virtual Reality and Gamification Within Procedurally Generated Environments to Increase Motivation During Gait Rehabilitation , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[22]  Steve Chi-Yin Yuen,et al.  Augmented Reality: An Overview and Five Directions for AR in Education , 2011 .

[23]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[24]  Gunther Notni,et al.  Comparing HMD-Based and Paper-Based Training , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[25]  Daniel Roggen,et al.  Exploring glass as a novel method for hands-free data entry in flexible cystoscopy , 2016, UbiComp.

[26]  Pengfei Zhou,et al.  Amateur: Augmented Reality Based Vehicle Navigation System , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[27]  Olivier Strauss,et al.  Ad-hoc Study on Soldiers Calibration Procedure in Virtual Reality , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[28]  Woontack Woo,et al.  The Effect of Avatar Appearance on Social Presence in an Augmented Reality Remote Collaboration , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[29]  Björn Eskofier,et al.  Evaluation of Interaction Techniques for a Virtual Reality Reading Room in Diagnostic Radiology , 2018, UIST.

[30]  Tara Akhavan,et al.  Towards Eye-Friendly VR: How Bright Should It Be? , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[31]  Peter Washington,et al.  Superpower glass: delivering unobtrusive real-time social cues in wearable systems , 2016, UbiComp Adjunct.

[32]  Mircea Teodorescu,et al.  Virtual Reality Video Game Paired with Physical Monocular Blurring as Accessible Therapy for Amblyopia , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[33]  Daniel Szafir,et al.  The Influence of Size in Augmented Reality Telepresence Avatars , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[34]  Gudrun Klinker,et al.  3D-FRC: Depiction of the future road course in the Head-Up-Display , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[35]  Jyh-Chong Liang,et al.  Current status, opportunities and challenges of augmented reality in education , 2013, Comput. Educ..

[36]  Andrew Y. C. Nee,et al.  Augmented reality applications in design and manufacturing , 2012 .

[37]  Robert LiKamWa,et al.  GLEAM -- An Illumination Estimation Framework for Real-time Photorealistic Augmented Reality on Mobile Devices (demo) , 2019, MobiSys.

[38]  Zhuoqing Morley Mao,et al.  QoE Doctor: Diagnosing Mobile App QoE with Automated UI Control and Cross-layer Analysis , 2014, Internet Measurement Conference.

[39]  Yasin Ozarslan,et al.  Augmented Reality in Education: Current Technologies and the Potential for Education , 2012 .

[40]  Tim Claudius Stratmann,et al.  Ensuring Safety in Augmented Reality from Trade-off Between Immersion and Situation Awareness , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[41]  Louis B. Rosenberg,et al.  The Use of Virtual Fixtures as Perceptual Overlays to Enhance Operator Performance in Remote Environments. , 1992 .

[42]  Shichang Xu,et al.  Mobilyzer: An Open Platform for Controllable Mobile Network Measurements , 2015, MobiSys.

[43]  Markku Turunen,et al.  The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[44]  Lei Xue,et al.  AndroidPerf: A cross-layer profiling system for Android applications , 2015, 2015 IEEE 23rd International Symposium on Quality of Service (IWQoS).

[45]  Nazlena Mohamad Ali,et al.  Evaluating Engagement Level and Analytical Support of Interactive Visualizations in Virtual Reality Environments , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[46]  Nassir Navab,et al.  Empirical Study of Non-Reversing Magic Mirrors for Augmented Reality Anatomy Learning , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[47]  Anind K. Dey,et al.  Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation , 2009, CHI.

[48]  Steven K. Feiner,et al.  The Effect of Narrow Field of View and Information Density on Visual Search Performance in Augmented Reality , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[49]  Judith Amores,et al.  TactileVR: Integrating Physical Toys into Learn and Play Virtual Reality Experiences , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[50]  Archan Misra,et al.  Empath-D: VR-based Empathetic App Design for Accessibility , 2018, MobiSys.