Towards Tangible Cultural Heritage Experiences—Enriching VR-based Object Inspection with Haptic Feedback

VR/AR technology is a key enabler for new ways of immersively experiencing cultural heritage artifacts based on their virtual counterparts obtained from a digitization process. In this article, we focus on enriching VR-based object inspection by additional haptic feedback, thereby creating tangible cultural heritage experiences. For this purpose, we present an approach for interactive and collaborative VR-based object inspection and annotation. Our system supports high-quality 3D models with accurate reflectance characteristics while additionally providing haptic feedback regarding shape features of the object based on a 3D printed replica. The digital object model in terms of a printable representation of the geometry as well as reflectance characteristics are stored in a compact and streamable representation on a central server, which streams the data to remotely connected users/clients. The latter can jointly perform an interactive inspection of the object in VR with additional haptic feedback through the 3D printed replica. Evaluations regarding system performance, visual quality of the considered models, as well as insights from a user study indicate an improved interaction, assessment, and experience of the considered objects.

[1]  Richard D. Joyce,et al.  Passive Haptics to Enhance Virtual Reality Simulations , 2017 .

[2]  Christopher Schwartz,et al.  Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage , 2011, VAST.

[3]  Roberto Scopigno,et al.  A compact representation of relightable images for the web , 2018, Web3D.

[4]  Vlastimil Havran,et al.  Lightdrum—Portable Light Stage for Accurate BTF Measurement on Site , 2017, Sensors.

[5]  Frederick P. Brooks,et al.  Moving objects in space: exploiting proprioception in virtual-environment interaction , 1997, SIGGRAPH.

[6]  Austin Hanus,et al.  A Collaborative Virtual Reality Escape Room with Passive Haptics , 2019, VR.

[7]  M. Koudelka,et al.  Acquisition , Compression , and Synthesis of Bidirectional Texture Functions , 2003 .

[8]  Kirk Martinez,et al.  Archaeological applications of polynomial texture mapping: analysis, conservation and representation , 2010 .

[9]  Weiliang Xu,et al.  Haptic display for virtual reality: progress and challenges , 2019, Virtual Real. Intell. Hardw..

[10]  Kyle Johnsen,et al.  Performance Benefits of High-Fidelity Passive Haptic Feedback in Virtual Reality Training , 2018, SUI.

[11]  Thomas Malzbender,et al.  New Reflection Transformation Imaging Methods for Rock Art and Multiple-Viewpoint Display , 2006, VAST.

[12]  Roberto Scopigno,et al.  A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions , 2012, Comput. Graph. Forum.

[13]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[14]  Holly E. Rushmeier,et al.  A Sparse Parametric Mixture Model for BTF Compression, Editing and Rendering , 2011, Comput. Graph. Forum.

[15]  Paul E. Debevec,et al.  A photometric approach to digitizing cultural artifacts , 2001, VAST '01.

[16]  Martin Doerr,et al.  Image-Based Empirical Information Acquisition, Scientific Reliability, and Long-Term Digital Preservation for the Natural Sciences and Cultural Heritage , 2008, Eurographics.

[17]  Didier Stricker,et al.  Fully Automatic, Omnidirectional Acquisition of Geometry and Appearance in the Context of Cultural Heritage Preservation , 2015, ACM Journal on Computing and Cultural Heritage.

[18]  Ramona Quattrini,et al.  Collaborative intelligence cyber-physical system for the valorization and re-use of cultural heritage , 2018, J. Inf. Technol. Constr..

[19]  Didier Stricker,et al.  A full-spherical device for simultaneous geometry and reflectance acquisition , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[20]  Christopher Schwartz,et al.  A Multi-camera, Multi-projector Super-Resolution Framework for Structured Light , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[21]  Tim Weyrich,et al.  Principles of Appearance Acquisition and Representation , 2009, Found. Trends Comput. Graph. Vis..

[22]  Michael Goesele,et al.  Advances in Geometry and Reflectance Acquisition , 2016, Eurographics.

[23]  Rafael Monroy,et al.  CultLab3D - On the Verge of 3D Mass Digitization , 2014, GCH.

[24]  Karol Myszkowski,et al.  Bidirectional Texture Function Compression Based on Multi‐Level Vector Quantization , 2010, Comput. Graph. Forum.

[25]  Andrea Giachetti,et al.  A novel framework for highlight reflectance transformation imaging , 2017, Comput. Vis. Image Underst..

[26]  Randy F. Pausch,et al.  Voodoo dolls: seamless interaction at multiple scales in virtual environments , 1999, SI3D.

[27]  Giljoo Nam,et al.  Practical SVBRDF acquisition of 3D objects with unstructured flash photography , 2018, ACM Trans. Graph..

[28]  Qinping Zhao,et al.  A survey on virtual reality , 2009, Science in China Series F: Information Sciences.

[29]  Caterina Balletti,et al.  TOOTEKO: A CASE STUDY OF AUGMENTED REALITY FOR AN ACCESSIBLE CULTURAL HERITAGE. DIGITIZATION, 3D PRINTING AND SENSORS FOR AN AUDIO-TACTILE EXPERIENCE , 2015 .

[30]  Michael Weinmann,et al.  OctreeBTFs - A compact, seamless and distortion-free reflectance representation , 2017, Comput. Graph..

[31]  Mitsuru Ishizuka,et al.  Enhancing Mediated Interpersonal Communication through Affective Haptics , 2009, INTETAIN.

[32]  Kasper Skou Ladefoged,et al.  Spatially-Varying Diffuse Reflectance Capture Using Irradiance Map Rendering for Image-Based Modeling Applications , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[33]  Andrea Giachetti,et al.  Neural reflectance transformation imaging , 2020, The Visual Computer.

[34]  Ernesto Damiani,et al.  Augmented reality technologies, systems and applications , 2010, Multimedia Tools and Applications.

[35]  G. Rainer,et al.  Neural BTF Compression and Interpolation , 2019, Comput. Graph. Forum.

[36]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[37]  Gero Müller,et al.  Data-driven methods for compression and editing of spatially varying appearance , 2008 .

[38]  Roberto Pierdicca,et al.  A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage , 2018, ACM Journal on Computing and Cultural Heritage.

[39]  Robert W. Lindeman,et al.  Hand-held windows: towards effective 2D interaction in immersive virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[40]  Paul Graham,et al.  Acquiring reflectance and shape from continuous spherical harmonic illumination , 2013, ACM Trans. Graph..

[41]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[42]  Vlastimil Havran,et al.  BRDF Slices: Accurate Adaptive Anisotropic Appearance Acquisition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Jon Gray,et al.  The Work of Art in the Age of Mechanical Reproduction , 2013 .

[44]  Paul E. Debevec,et al.  A dual light stage , 2005, EGSR '05.

[45]  Paul E. Debevec,et al.  Image-based lighting , 2002, IEEE Computer Graphics and Applications.

[46]  Christopher Schwartz,et al.  Level‐of‐Detail Streaming and Rendering using Bidirectional Sparse Virtual Texture Functions , 2013, Comput. Graph. Forum.

[47]  Christopher Schwartz,et al.  Design and Implementation of Practical Bidirectional Texture Function Measurement Devices Focusing on the Developments at the University of Bonn , 2014, Sensors.

[48]  Jirí Filip,et al.  Visual Texture , 2013, Advances in Computer Vision and Pattern Recognition.

[49]  Christopher Schwartz,et al.  WebGL-based streaming and presentation of objects with bidirectional texture functions , 2013, JOCCH.

[50]  Zen-Chung Shih,et al.  Modeling Bidirectional Texture Functions with Multivariate Spherical Radial Basis Functions , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Reinhard Klein,et al.  BTF Compression via Sparse Tensor Decomposition , 2009, Comput. Graph. Forum.

[52]  Michal Haindl,et al.  Visual Texture: Accurate Material Appearance Measurement, Representation and Modeling , 2013 .

[53]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[54]  Todd E. Zickler,et al.  A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance , 2010, ACM Trans. Graph..

[55]  Takuji Narumi,et al.  Ascending and Descending in Virtual Reality: Simple and Safe System Using Passive Haptics , 2018, IEEE Transactions on Visualization and Computer Graphics.

[56]  Tim Weyrich,et al.  Unified Neural Encoding of BTFs , 2020, Comput. Graph. Forum.

[57]  Jirí Filip,et al.  Extreme Compression and Modeling of Bidirectional Texture Function , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Didier Stricker,et al.  Faithful, compact and complete digitization of cultural heritage using a full-spherical scanner , 2013, 2013 Digital Heritage International Congress (DigitalHeritage).

[59]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .