Pick-by-vision: bringing HMD-based augmented reality into the warehouse

The technology of Augmented Reality exists for several decades and it is attributed the potential to provide an ideal, efficient and intuitive way of presenting information. However it is not yet widely used. This is because the realization of such Augmented Reality systems requires to solve many principal problems from various areas. Much progress has been made in solving problems of core technologies, which enables us now to intensively explore the development of Augmented Reality applications. As an exemplary industrial use case for this exploration, I selected the order picking process in logistics applications. This thesis reports on the development of an application to support this task, by iteratively improving Augmented Reality-based metaphors. In such order picking tasks, workers collect sets of items from assortments in warehouses according to work orders. This order picking process has been subject to optimization for a long time, as it occurs a million times a day in industrial life. For this Augmented Reality application development, workers have been equipped with mobile hardware, consisting of a wearable computer (in a back-pack) and tracked head-mounted displays (HMDs). This thesis presents the iterative approach of exploring, evaluating and refining the Augmented Reality system, focusing on usability and utility. It starts in a simple laboratory setup and goes up to a realistic industrial setup in a factory hall. The Augmented Reality visualization shown in the HMD was the main subject of optimization in this thesis. Overall, the task was challenging, as workers have to be guided on different levels, from very coarse to very fine granularity and accuracy. The resulting setup consists of a combined and adaptive visualization to precisely and efficiently guide the user, even if the actual target of the augmentation is not always in the field of view of the HMD. A side-effect of this iterative evaluation and refinement of visualizations in an industrial setup is the report on many lessons learned and an advice on the way Augmented Reality user interfaces should be improved and refined.

[1]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[2]  W. Hofmann,et al.  Quantitative Methoden : Einführung in die Statistik für Psychologen und Sozialwissenschaftler , 2010 .

[3]  Gudrun Klinker,et al.  Effective control of a car driver's attention for visual and acoustic guidance towards the direction of imminent dangers , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[4]  Bernd Brügge,et al.  Design of a component-based augmented reality framework , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[5]  Willibald A. Günthner,et al.  Pick-by-Vision: Augmented Reality unterstützte Kommissionierung , 2009 .

[6]  Doug A. Bowman,et al.  Formalizing the Design, Evaluation, and Application of Interaction Techniques for Immersive Virtual Environments , 1999, J. Vis. Lang. Comput..

[7]  D. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[8]  Jared M. Spool,et al.  Testing web sites: five users is nowhere near enough , 2001, CHI Extended Abstracts.

[9]  J. Edward Swan,et al.  Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design , 2008, IEEE Transactions on Visualization and Computer Graphics.

[10]  Gudrun Klinker,et al.  TUM-I 0819 : An Evaluation of Augmented Reality Visualizations to Support the Order Picking , 2008 .

[11]  Rüdiger Mecke,et al.  Distribution of visual attention in head-worn displays , 2008, APGV '08.

[12]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[13]  Ben Shneiderman,et al.  Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis , 2008, CHI.

[14]  Deborah Hix,et al.  User-Centered Design and Evaluation of Virtual Environments , 1999, IEEE Computer Graphics and Applications.

[15]  Steven K. Feiner,et al.  Situated documentaries: embedding multimedia presentations in the real world , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[16]  Sheng Liu,et al.  An optical see-through head mounted display with addressable focal planes , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[17]  Gudrun Klinker,et al.  A System Architecture for Ubiquitous Tracking Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[18]  Dan R. Olsen,et al.  Evaluating user interface systems research , 2007, UIST.

[19]  Robert S. Laramee,et al.  Rivalry and interference with a head-mounted display , 2002, TCHI.

[20]  Randall E. Bailey,et al.  Pathway design effects on synthetic vision head-up displays , 2004, SPIE Defense + Commercial Sensing.

[21]  Randolph G. Bias,et al.  Cost-Justifying Usability: An Update for the Internet Age , 2005 .

[22]  Leela Damodaran,et al.  User involvement in the systems design process-a practical guide for users , 1996, Behav. Inf. Technol..

[23]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[24]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[25]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[26]  M. Pauline Baker,et al.  Computer Graphics, C Version , 1996 .

[27]  Holger Regenbrecht,et al.  Augmented reality projects in the automotive and aerospace industries , 2005, IEEE Computer Graphics and Applications.

[28]  Wilhelm Dangelmaier,et al.  AR Support for Picking , 2005, SimVis.

[29]  Woodrow Barfield,et al.  Boeing’s Wire Bundle Assembly Project , 2001 .

[30]  Deborah Hix,et al.  Usability engineering: domain analysis activities for augmented-reality systems , 2002, IS&T/SPIE Electronic Imaging.

[31]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[32]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[33]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[34]  Deborah Hix,et al.  User-centered design and evaluation of a real-time battlefield visualization virtual environment , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[35]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[36]  Noam Tractinsky,et al.  Aesthetics and apparent usability: empirically assessing cultural and methodological issues , 1997, CHI.

[37]  Christian Sandor,et al.  Experimental evaluation of an augmented reality visualization for directing a car driver's attention , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[38]  Nassir Navab,et al.  Practical solutions for calibration of optical see-through devices , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[39]  Alan M. Davis,et al.  Requirements elicitation and elicitation technique selection: model for two knowledge-intensive software development processes , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[40]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[41]  M. Scriven.  Evaluation thesaurus, 4th ed. , 1991 .

[42]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[43]  Martin Bauer.  Tracking errors in augmented reality , 2007 .

[44]  Steven K. Feiner,et al.  Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[45]  Jens Grubert,et al.  Perceptual issues in optical-see-through displays , 2010, APGV '10.

[46]  Willibald A. Günthner,et al.  Pick-by-Vision comes on age: evaluation of an augmented reality supported picking system in a real storage environment , 2009, AFRIGRAPH '09.

[47]  Jannick P. Rolland,et al.  Towards Quantifying Depth and Size Perception in Virtual Environments , 1993, Presence: Teleoperators & Virtual Environments.

[48]  Olaf Oehme.  Ergonomische Untersuchung von kopfbasierten Displays für Anwendungen der erweiterten Realität in Produktion und Service , 2004 .

[49]  Nassir Navab,et al.  Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[50]  A. Sutcliffe.  User-Centered Requirements Engineering , 2002 .

[51]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[52]  Suzanne Robertson,et al.  Requirements trawling: techniques for discovering requirements , 2001, Int. J. Hum. Comput. Stud..

[53]  Jodi Forlizzi,et al.  Understanding experience in interactive systems , 2004, DIS '04.

[54]  Gitte Lindgaard,et al.  Usability testing: what have we overlooked? , 2007, CHI.

[55]  Mats I. Johansson,et al.  Design and performance of kitting and order picking systems , 1995 .

[56]  Paul M. Sharkey,et al.  Identifying Likely Successful Users of Virtual Reality Systems , 1998, Presence.

[57]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[58]  M. Scriven.  The methodology of evaluation , 1966 .

[59]  Katharina Pentenrieder,et al.  Augmented Reality based Factory Planning , 2009 .

[60]  Mark Billinghurst,et al.  Applying HCI principles to AR systems design , 2007 .

[61]  Wilhelm Dangelmaier,et al.  Augmented Reality applications for Warehouse Logistics , 2005, WSTST.

[62]  Patrick Baudisch,et al.  Halo: a Technique for Visualizing Off-Screen Locations , 2003 .

[63]  Mark A. Livingston,et al.  Evaluating Human Factors in Augmented Reality Systems , 2005, IEEE Computer Graphics and Applications.

[64]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[65]  J. Bortz.  Statistik für Human- und Sozialwissenschaftler , 2010 .

[66]  C. Osgood,et al.  Semantic differential technique : a sourcebook , 1969 .

[67]  Hiroshi Ishii,et al.  Illuminating light: an optical design tool with a luminous-tangible interface , 1998, CHI.

[68]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[69]  Gudrun Klinker,et al.  Supporting order picking with Augmented Reality , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[70]  Sari Kujala,et al.  User involvement: A review of the benefits and challenges , 2003, Behav. Inf. Technol..

[71]  Dieter Schmalstieg,et al.  Location based Applications for Mobile Augmented Reality , 2003, AUIC.

[72]  Anker Helms Jørgensen,et al.  Thinking-aloud in user interface design: a method promoting cognitive ergonomics , 1990 .

[73]  P. Kidwell,et al.  The trouble with computers: Usefulness, usability and productivity , 1996, IEEE Annals of the History of Computing.

[74]  Rüdiger Mecke,et al.  Mobile Augmented Reality in industrial applications: Approaches for solution of user-related issues , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[75]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[76]  Shih-Ching Yeh,et al.  Development of a Benchmarking Scenario for Testing 3D User Interface Devices and Interaction Methods , 2005 .

[77]  Frank Biocca,et al.  Attention funnel: omnidirectional 3D cursor for mobile augmented reality platforms , 2006, CHI.

[78]  Bonnie E. John,et al.  Two Case Studies in Using Cognitive Walkthrough for Interface Evaluation , 2000 .

[79]  Deborah Hix,et al.  Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2001 .

[80]  Paul J. Sherman.  Usability Success Stories: How Organizations Improve by Making Easier-to-use Software And Web Sites , 2006 .

[81]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[82]  Mark Billinghurst,et al.  A survey of evaluation techniques used in augmented reality studies , 2008, SIGGRAPH 2008.

[83]  Thomas Alt.  Augmented Reality in der Produktion , 2003 .

[84]  F. Heider.  The psychology of interpersonal relations , 1958 .

[85]  Debbie Stone,et al.  User Interface Design and Evaluation , 2005 .

[86]  Dennis R. Wixon.  Evaluating usability methods: why the current literature fails the practitioner , 2003, INTR.

[87]  Gudrun Klinker,et al.  The Intelligent Welding Gun: Augmented Reality for Experimental Vehicle Construction , 2004 .

[88]  Fan Xiao,et al.  Display-relative calibration for optical see-through head-mounted displays , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[89]  Alan M. Davis,et al.  Elicitation technique selection: how do experts do it? , 2003, Proceedings. 11th IEEE International Requirements Engineering Conference, 2003..

[90]  Margarita Anastassova,et al.  Prototype Evaluation and User-Needs Analysis in the Early Design of Emerging Technologies , 2007, HCI.

[91]  Johannes Tümler.  Untersuchungen zu nutzerbezogenen und technischen Aspekten beim Langzeiteinsatz mobiler Augmented Reality Systeme in industriellen Anwendungen , 2009 .

[92]  Andreas Butz,et al.  SearchLight - A Lightweight Search Function for Pervasive Environments , 2004, Pervasive.

[93]  Gudrun Klinker,et al.  Mobile Information Presentation Schemes for Supra-adaptive Logistics Applications , 2006, ICAT.

[94]  B. Schneirdeman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[95]  Jakob Nielsen,et al.  Usability inspection methods: executive summary , 1995 .

[96]  Masaaki Kurosu,et al.  Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability , 1995, CHI 95 Conference Companion.

[97]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[98]  E. Kuntsche,et al.  Multivariate Analysemethoden: Eine anwendungsorientierte Einführung. , 2001 .

[99]  Gudrun Klinker,et al.  Using laser projectors for augmented reality , 2008, VRST '08.

[100]  Willibald A. Günthner.  Neue Wege in der Automobillogistik , 2007 .

[101]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[102]  Tobias Constantin Haupt.  Persönlichkeitstyp und Stresserleben , 2004 .

[103]  L. Faulkner.  Beyond the five-user assumption: Benefits of increased sample sizes in usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[104]  Tobias Höllerer,et al.  A cost-effective usability evaluation progression for novel interactive systems , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[105]  Paul Milgram,et al.  Perceptual issues in augmented reality , 1996, Electronic Imaging.

[106]  Kasper Hornbæk,et al.  Two psychology-based usability inspection techniques studied in a diary experiment , 2004, NordiCHI '04.

[107]  Bernard D. Adelstein,et al.  Conducting Human-Subject Experiments with Virtual and Augmented Reality , 2006 .

[108]  Gavriel Salvendy,et al.  Effectiveness of user testing and heuristic evaluation as a function of performance classification , 2002, Behav. Inf. Technol..

[109]  Robert C. Williges,et al.  An Evaluation of Critical Incidents for Software Documentation Design , 1986 .

[110]  Michael Burmester,et al.  Der User Experience (UX) auf der Spur: Zum Einsatz von www.attrakdiff.de , 2008, Usability Professionals.