MRCAT: In Situ Prototyping of Interactive AR Environments

Augmented reality (AR) blends physical and virtual components to create a mixed reality experience. This unique display medium presents new opportunities for application design, as applications can move beyond the desktop and integrate with the physical environment. In order to build effective applications for AR displays, we need to be able to iteratively design for different contexts or scenarios. We present MRCAT (Mixed Reality Content Authoring Toolkit), a tool for in situ prototyping of mixed reality environments. We discuss the initial design of MRCAT and iteration after a study (\(N = 14\)) to evaluate users’ abilities to craft AR applications with MRCAT and with a 2D prototyping tool. We contextualize our system in a case study of museum exhibit development, identifying how existing ideation and prototyping workflows could be bolstered with the approach offered by MRCAT. With our exploration of in situ prototyping, we enumerate key aspects both of AR application design and targeted domains that help guide design of more effective AR prototyping tools.

[1]  Sukhyun Lim,et al.  An augmented reality-based authoring tool for E-learning applications , 2011, Multimedia Tools and Applications.

[2]  Xiang 'Anthony' Chen,et al.  Typing on Glasses: Adapting Text Entry to Smart Eyewear , 2015, MobileHCI.

[3]  Gang Ren,et al.  SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing , 2017, UIST.

[4]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[5]  W. Mackay,et al.  Prototyping tools and techniques , 2002 .

[6]  Werner Hartmann,et al.  Authoring of a mixed reality assembly instructor for hierarchical structures , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[7]  J. Preece,et al.  The Human-Computer Interaction Handbook , 2003 .

[8]  Steven K. Feiner,et al.  WeARHand: Head-worn, RGB-D camera-based, bare-hand user interface with visually enhanced depth perception , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[9]  Robin Vivian,et al.  CARDS: A Mixed-Reality System for Collaborative Learning at School , 2019, ISS.

[10]  Hai-Ning Liang,et al.  Errata to "RingText: Dwell-Free and Hands-Free Text Entry for Mobile Head-Mounted Displays Using Head Motions" , 2019, IEEE Trans. Vis. Comput. Graph..

[11]  Tobias Höllerer,et al.  Evaluating gesture-based augmented reality annotation , 2017, 2017 IEEE Symposium on 3D User Interfaces (3DUI).

[12]  Beat Signer,et al.  SpeeG: a multimodal speech- and gesture-based text input solution , 2012, AVI.

[13]  Mark Billinghurst,et al.  ComposAR: An intuitive tool for authoring AR applications , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[14]  Jae Yeol Lee,et al.  CAPTIVE: a cube with augmented physical tools , 2014, CHI Extended Abstracts.

[15]  Albrecht Schmidt,et al.  FrameBox and MirrorBox: tools and guidelines to support designers in prototyping interfaces for 3D displays , 2014, CHI.

[16]  Gerard Jounghyun Kim,et al.  Immersive authoring of tangible augmented reality applications , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[17]  Mark Billinghurst,et al.  Mixed reality collaboration through sharing a live panorama , 2017, SIGGRAPH ASIA Mobile Graphics and Interactive Applications.

[18]  Woontack Woo,et al.  A usability study of multimodal input in an augmented reality environment , 2013, Virtual Reality.

[19]  Julia Woodward,et al.  Experimental Analysis of Single Mode Switching Techniques in Augmented Reality , 2019, Graphics Interface.

[20]  Hooman Hedayati,et al.  Robot Teleoperation with Augmented Reality Virtual Surrogates , 2019, 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[21]  David Kim,et al.  Looking Inside the Wires: Understanding Museum Visitor Learning with an Augmented Circuit Exhibit , 2017, CHI.

[22]  Michael J. Muller,et al.  Participatory design: the third space in HCI , 2002 .

[23]  Hirokazu Kato,et al.  Augmented Reality Learning Experiences: Survey of Prototype Design and Evaluation , 2014, IEEE Transactions on Learning Technologies.

[24]  Krzysztof Walczak,et al.  Building Virtual and Augmented Reality museum exhibitions , 2004, Web3D '04.

[25]  J. Antonio García-Macías,et al.  Browsing the Internet of Things with Sentient Visors , 2011, Computer.

[26]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[27]  Mark Billinghurst,et al.  Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[28]  Carlos Delgado Kloos,et al.  Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness , 2014, Comput. Educ..

[29]  Andy Cockburn,et al.  User-defined gestures for augmented reality , 2013, INTERACT.

[30]  Krzysztof Walczak,et al.  Creation of Interactive AR Content on Mobile Devices , 2013, BIS.

[31]  Chris North,et al.  Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures , 2011, Int. J. Hum. Comput. Stud..

[32]  Sriram Karthik Badam,et al.  Affordances of Input Modalities for Visual Data Exploration in Immersive Environments , 2017 .

[33]  Steven K. Feiner,et al.  Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality , 2003, ICMI '03.

[34]  Yuanchun Shi,et al.  Tap, Dwell or Gesture?: Exploring Head-Based Text Entry Techniques for HMDs , 2017, CHI.

[35]  Rubaiat Habib Kazi,et al.  SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ , 2018, CHI.

[36]  Christopher D. Hundhausen,et al.  How is User Interface Prototyping Really Done in Practice? A Survey of User Interface Designers , 2010, 2010 IEEE Symposium on Visual Languages and Human-Centric Computing.

[37]  Raimund Dachselt,et al.  Three-dimensional menus: A survey and taxonomy , 2007, Comput. Graph..

[38]  Roger R. Hall,et al.  Prototyping for usability of new technology , 2001, Int. J. Hum. Comput. Stud..

[39]  Tobias Höllerer,et al.  An Evaluation of Bimanual Gestures on the Microsoft HoloLens , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[40]  Elizabeth F. Churchill,et al.  Mobile augmented reality: exploring design and prototyping techniques , 2012, Mobile HCI.

[41]  Mark Billinghurst,et al.  A Wizard of Oz study for an AR multimodal interface , 2008, ICMI '08.

[42]  Per Ola Kristensson,et al.  Performance Envelopes of in-Air Direct and Smartwatch Indirect Control for Head-Mounted Augmented Reality , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[43]  Kevin Ponto,et al.  Designing extreme 3D user interfaces for augmented live performances , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[44]  Krzysztof Walczak,et al.  Dynamic creation of interactive mixed reality presentations , 2005, VRST '05.

[45]  Tobias Höllerer,et al.  Gesture-based augmented reality annotation , 2017, 2017 IEEE Virtual Reality (VR).

[46]  Michael J. McGuffin,et al.  DualCAD: Integrating Augmented Reality with a Desktop GUI and Smartphone Interaction , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[47]  Ajune Wanis Ismail,et al.  Ar home deco: virtual object manipulation technique using hand gesture in augmented reality , 2018 .

[48]  Rosanna Maria Viglialoro,et al.  A Microsoft HoloLens Mixed Reality Surgical Simulator for Patient-Specific Hip Arthroplasty Training , 2018, AVR.

[49]  Blair MacIntyre,et al.  DART: a toolkit for rapid design exploration of augmented reality experiences , 2005, ACM Trans. Graph..

[50]  Heedong Ko,et al.  "Move the couch where?" : developing an augmented reality multimodal interface , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[51]  Christian R. Prause,et al.  The Energy Aware Smart Home , 2010, 2010 5th International Conference on Future Information Technology.

[52]  Danielle Albers Szafir,et al.  Interacting with Distant Objects in Augmented Reality , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).