Visual Coherence in Mixed Reality: A Systematic Enquiry

Virtual and augmented reality, and other forms of mixed reality (MR), have become a focus of attention for companies and researchers. Before they can become successful in the market and in society, those MR systems must be able to deliver a convincing, novel experience for the users. By definition, the experience of mixed reality relies on the perceptually successful blending of reality and virtuality. Any MR system has to provide a sensory, in particular visually coherent, set of stimuli. Therefore, issues with visual coherence, that is, a discontinued experience of a MR environment, must be avoided. While it is very easy for a user to detect issues with visual coherence, it is very difficult to design and implement a system for coherence. This article presents a framework and exemplary implementation of a systematic enquiry into issues with visual coherence and possible solutions to address those issues. The focus is set on head-mounted display-based systems, notwithstanding its applicability to other types of MR systems. Our framework, together with a systematic discussion of tangible issues and solutions for visual coherence, aims at guiding developers of mixed reality systems for better and more effective user experiences.

[1]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[2]  Erik Viirre,et al.  The Virtual Retinal Display: A new Display Technology using Scanned Laser Light , 1998 .

[3]  Dirk Bartz,et al.  Occlusion handling for medical augmented reality using a volumetric phantom model , 2004, VRST '04.

[4]  Andrew W. Fitzgibbon,et al.  Spatial calibration of an optical see-through head-mounted display , 2008, Journal of Neuroscience Methods.

[5]  Bruce H. Thomas,et al.  A wearable computer system with augmented reality to support terrestrial navigation , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[6]  Alan Schwartz,et al.  Practice on an Augmented Reality/Haptic Simulator and Library of Virtual Brains Improves Residents’ Ability to Perform a Ventriculostomy , 2013, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[7]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[8]  Hakan Urey,et al.  State of the Art in Stereoscopic and Autostereoscopic Displays , 2011, Proceedings of the IEEE.

[9]  Naokazu Yokoya,et al.  A stereoscopic video see-through augmented reality system based on real-time vision-based registration , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[10]  Werner Hartmann,et al.  A real-time shadow approach for an augmented reality application using shadow volumes , 2003, VRST '03.

[11]  Gudrun Klinker,et al.  The Intelligent Welding Gun: Augmented Reality for Experimental Vehicle Construction , 2004 .

[12]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[13]  Reinhard Koch,et al.  Time-of-Flight sensor calibration for accurate range sensing , 2010, Comput. Vis. Image Underst..

[14]  Fan Xiao,et al.  Display-relative calibration for optical see-through head-mounted displays , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[15]  Daniel Herrera C,et al.  Joint depth and color camera calibration with distortion correction. , 2012, IEEE transactions on pattern analysis and machine intelligence.

[16]  Naokazu Yokoya,et al.  Real-time estimation of light source environment for photorealistic augmented reality , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[17]  Eric Foxlin,et al.  Encoded LED system for optical trackers , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[18]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[19]  Kazutaka Inoguchi,et al.  Thin wide-field-of-view HMD with free-form-surface prism and applications , 1999, Electronic Imaging.

[20]  Karol Myszkowski,et al.  Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Holger Regenbrecht,et al.  A leap-supported, hybrid AR interface approach , 2013, OZCHI.

[22]  Matthias M. Wloka,et al.  Resolving occlusion in augmented reality , 1995, I3D '95.

[23]  J. Hakkinen,et al.  Postural stability and sickness symptoms after HMD use , 2002, IEEE International Conference on Systems, Man and Cybernetics.

[24]  Zhengyou Zhang,et al.  Calibration between depth and color sensors for commodity depth cameras , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[25]  Frank Biocca,et al.  Development of Head-Mounted Projection Displays for Distributed, Collaborative, Augmented Reality Applications , 2005, Presence: Teleoperators & Virtual Environments.

[26]  Holger Regenbrecht,et al.  Visual manipulations for motor rehabilitation , 2012, Comput. Graph..

[27]  Hong Hua,et al.  Occlusion capable optical see-through head-mounted display using freeform optics , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[28]  Holger Regenbrecht,et al.  Augmented reality projects in the automotive and aerospace industries , 2005, IEEE Computer Graphics and Applications.

[29]  Ulrich Neumann,et al.  Vision-based pose computation: robust and accurate augmented reality tracking , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[30]  Li Li,et al.  Photorealistic rendering for augmented reality using environment illumination , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[31]  Steven K. Feiner,et al.  Perceptual issues in augmented reality revisited , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[32]  Jong-Il Park,et al.  Invisible marker tracking for AR , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[33]  Hong Hua,et al.  High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics. , 2014, Optics express.

[34]  Sarah Sharples,et al.  Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems , 2008, Displays.

[35]  Martin Knecht,et al.  Differential Instant Radiosity for mixed reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[36]  Yoichi Sato,et al.  Real-Time Fingertip Tracking and Gesture Recognition , 2002, IEEE Computer Graphics and Applications.

[37]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion of real and virtual environments , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[38]  Sheng Liu,et al.  A Novel Prototype for an Optical See-Through Head-Mounted Display with Addressable Focus Cues , 2010, IEEE Transactions on Visualization and Computer Graphics.

[39]  Pourang Irani,et al.  SmartColor: Real-time color correction and contrast for optical see-through head-mounted displays , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[40]  Kiyoshi Kiyokawa,et al.  Fisheye vision: peripheral spatial compression for improved field of view in head mounted displays , 2014, SUI.

[41]  Holger Regenbrecht,et al.  Beyond the Looking Glass: Fooling the Brain with the Augmented Mirror Box , 2011, PRESENCE: Teleoperators and Virtual Environments.

[42]  Andrew I. Comport,et al.  Robust and real-time image-based tracking for markerless augmented reality , 2003 .

[43]  Dieter Schmalstieg,et al.  Efficient and robust radiance transfer for probeless photorealistic augmented reality , 2014, 2014 IEEE Virtual Reality (VR).

[44]  Takeshi Oishi,et al.  Visibility-based blending for real-time applications , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[45]  Nassir Navab,et al.  Single-Point Active Alignment Method (SPAAM) for Optical See-Through HMD Calibration for Augmented Reality , 2002, Presence: Teleoperators & Virtual Environments.

[46]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[47]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[48]  Naokazu Yokoya,et al.  Real-time composition of stereo images for video see-through augmented reality , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[49]  Céline Loscos,et al.  Classification of Illumination Methods for Mixed Reality , 2006, Comput. Graph. Forum.

[50]  Marie-Odile Berger.  Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[51]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[52]  Olivier Bau,et al.  REVEL: tactile feedback technology for augmented reality , 2012, ACM Trans. Graph..

[53]  Franklin C. Crow,et al.  Shadow algorithms for computer graphics , 1977, SIGGRAPH.

[54]  Céline Loscos,et al.  Automatic generation of consistent shadows for augmented reality , 2005, Graphics Interface.

[55]  David W. Murray,et al.  Compositing for small cameras , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[56]  M. M. Shah,et al.  Occlusion in augmented reality , 2012, 2012 8th International Conference on Information Science and Digital Content Technology (ICIDT2012).

[57]  Mark Billinghurst,et al.  An occlusion capable optical see-through head mount display for supporting co-located collaboration , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[58]  Holger Regenbrecht,et al.  Real-Time Radiometric Compensation for Optical See-Through Head-Mounted Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[59]  Hannes Kaufmann,et al.  High-quality reflections, refractions, and caustics in Augmented Reality and their contribution to visual coherence , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[60]  Greg Welch,et al.  Shader Lamps: Animating Real Objects With Image-Based Illumination , 2001, Rendering Techniques.

[61]  Dieter Schmalstieg,et al.  Finger tracking for interaction in augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[62]  Pierre Fite-Georgel,et al.  Is there a reality in Industrial Augmented Reality? , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.