A usability scale for handheld augmented reality

Handheld augmented reality (HAR) applications must be carefully designed and improved based on user feedback to sustain commercial use. However, no standard questionnaire considers perceptual and ergonomic issues found in HAR. We address this issue by creating a HAR Usability Scale (HARUS). To create HARUS, we performed a systematic literature review to enumerate user-reported issues in HAR applications. Based on these issues, we created a questionnaire measuring manipulability -- the ease of handling the HAR system, and comprehensibility -- the ease of understanding the information presented by HAR. We then provide evidences of validity and reliability of the HARUS questionnaire by applying it to three experiments. The results show that HARUS consistently correlates with other subjective and objective measures of usability, thereby supporting its concurrent validity. Moreover, HARUS obtained a good Cronbach's alpha in all three experiments, thereby demonstrating internally consistency. HARUS, as well as its decomposition into individual manipulability and comprehensibility scores, are evaluation tools that researchers and professionals can use to analyze their HAR applications. By providing such a tool, they can gain quality feedback from users to improve their HAR applications towards commercial success.

[1]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[2]  D. Dillman,et al.  International handbook of survey methodology. , 2008 .

[3]  Dieter Schmalstieg,et al.  Anywhere Interfaces Using Handheld Augmented Reality , 2012, Computer.

[4]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[5]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[6]  Thomas Olsson,et al.  Online user survey on current mobile augmented reality applications , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[7]  James D. Wright,et al.  Handbook of Survey Research. , 1985 .

[8]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[9]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[10]  Samuel Messick,et al.  VALIDITY OF TEST INTERPRETATION AND USE , 1990 .

[11]  Eeva M. Pilke Flow experiences in information technology use , 2004, Int. J. Hum. Comput. Stud..

[12]  S. Kurkovsky,et al.  Current issues in handheld augmented reality , 2012, 2012 International Conference on Communications and Information Technology (ICCIT).

[13]  Mark Billinghurst,et al.  Exploring the use of handheld AR for outdoor navigation , 2012, Comput. Graph..

[14]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[15]  Tonya L. Smith-Jackson,et al.  Reliability and validity of the mobile phone usability questionnaire (MPUQ) , 2006 .

[16]  Dieter Schmalstieg,et al.  User experiences with augmented reality aided navigation on phones , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[17]  J. Edward Swan,et al.  Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design , 2008, IEEE Transactions on Visualization and Computer Graphics.

[18]  Dieter Schmalstieg,et al.  Handheld Augmented Reality for underground infrastructure visualization , 2008, Personal and Ubiquitous Computing.

[19]  Mark Billinghurst,et al.  A survey of evaluation techniques used in augmented reality studies , 2008, SIGGRAPH 2008.

[20]  Chris Dede,et al.  EcoMOBILE: Integrating augmented reality and probeware with environmental education field trips , 2013, Comput. Educ..

[21]  Jun Rekimoto A Magnifying Glass Approach to Augmented Reality Systems , 1997, Presence Teleoperators Virtual Environ..

[22]  Eduardo E. Veas,et al.  Vesp’R: design and evaluation of a handheld AR device , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[23]  Heidi A. Diefes-Dux,et al.  A preliminary validation of Attention, Relevance, Confidence and Satisfaction model-based Instructional Material Motivational Survey in a computer-based tutorial setting , 2006, Br. J. Educ. Technol..

[24]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[25]  Tobias Höllerer,et al.  A cost-effective usability evaluation progression for novel interactive systems , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[26]  A. Dunser,et al.  CityViewAR: A mobile outdoor AR application for city visualization , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media, and Humanities (ISMAR-AMH).

[27]  Sean White,et al.  SiteLens: situated visualization techniques for urban site visits , 2009, CHI.

[28]  P. V. Marsden,et al.  Handbook of Survey Research , 1985 .

[29]  Dieter Schmalstieg,et al.  Mobile augmented reality for environmental monitoring , 2012, Personal and Ubiquitous Computing.

[30]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[31]  R. Radhakrishna Tips for Developing and Testing Questionnaires/Instruments , 2007 .

[32]  Arindam Dey,et al.  Lessons learned: Evaluating visualizations for occluded objects in handheld augmented reality , 2014, Int. J. Hum. Comput. Stud..

[33]  Steven K. Feiner,et al.  Perceptual issues in augmented reality revisited , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[34]  Hirokazu Kato,et al.  Augmented Reality Learning Experiences: Survey of Prototype Design and Evaluation , 2014, IEEE Transactions on Learning Technologies.

[35]  Deborah Hix,et al.  Perceptual and Ergonomic Issues in Mobile Augmented Reality for Urban Operations , 2003 .

[36]  Dieter Schmalstieg,et al.  Sketching up the world: in situ authoring for mobile Augmented Reality , 2012, Personal and Ubiquitous Computing.

[37]  Leonard S. Cahen,et al.  Educational Testing Service , 1970 .

[38]  Eduardo E. Veas,et al.  Handheld devices for mobile augmented reality , 2010, MUM.

[39]  Dieter Schmalstieg,et al.  Handheld augmented reality indoor navigation with activity-based instructions , 2011, Mobile HCI.