Reproducing Real World Acoustics in Virtual Reality UsingSpherical Cameras

Virtual Reality (VR) systems have been intensely explored, with several research communities investigating the different modalities involved. Regarding the audio modality, one of the main issues is the generation of sound that is perceptually coherent with the visual reproduction. Here, we propose a pipeline for creating plausible interactive reverb using visual information: first, we characterize real environment acoustics given a pair of spherical cameras; then, we reproduce reverberant spatial sound, by using the estimated acoustics, within a VR scene. The evaluation is made by extracting the room impulse responses (RIRs) of four virtually rendered rooms. Results show agreement, in terms of objective metrics, between the synthesized acoustics and the ones calculated from RIRs recorded within the respective real rooms.

[1]  L. Milling,et al.  The effectiveness of virtual reality distraction for pain reduction: a systematic review. , 2010, Clinical psychology review.

[2]  Adrian Hilton,et al.  3D Room Geometry Reconstruction Using Audio-Visual Sensors , 2017, 2017 International Conference on 3D Vision (3DV).

[3]  J. S. Bradley,et al.  Review of objective room acoustics measures and future needs , 2011 .

[4]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Philip J. B. Jackson,et al.  Estimation of Room Reflection Parameters for a Reverberant Spatial Audio Object , 2015 .

[6]  J. Rix,et al.  Virtual prototyping : virtual environments and the product design process : proceedings of the IFIP WG 5.10 workshops on virtual environments and their applications and virtual prototyping, 1994 , 1995 .

[7]  Guy-Bart Stan,et al.  Comparison of different impulse response measurement techniques , 2002 .

[8]  Sebastian J. Schlecht,et al.  Audio Quality Evaluation in Virtual Reality: Multiple Stimulus Ranking with Behavior Tracking , 2018 .

[9]  Frank Melchior,et al.  Object-Based Reverberation for Spatial Audio , 2017 .

[10]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Adrian Hilton,et al.  Volumetric performance capture from minimal camera viewpoints , 2018, ECCV.

[12]  Honglak Lee,et al.  Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision , 2016, NIPS.

[13]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[14]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[15]  Daniel A. Guttentag.  Virtual reality: Applications and implications for tourism , 2010 .

[16]  Bruno Fazenda,et al.  The Effect of Visual Cues and Binaural Rendering Method on Plausibility in Virtual Environments , 2018 .

[17]  Carsten Rother,et al.  Dense Semantic Image Segmentation with Objects and Attributes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Neville H Fletcher,et al.  Springer handbook of acoustics (2nd edition) , 2014 .

[19]  Xiaofeng Ren,et al.  Toward Robust Material Recognition for Everyday Objects , 2011, BMVC.

[20]  Angelo Farina,et al.  Simultaneous Measurement of Impulse Response and Distortion with a Swept-Sine Technique , 2000 .

[21]  Kwanghoon Sohn,et al.  3D reconstruction from stereo images for interactions between real and virtual objects , 2005, Signal Process. Image Commun..

[22]  Eric G. Johnson,et al.  The effect of virtual reality gaming on dynamic balance in older adults. , 2012, Age and ageing.

[23]  Soon-Wook Kwon,et al.  Fitting range data to primitives for rapid local 3D modeling using sparse range point clouds , 2004 .

[24]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[25]  Soh-Khim Ong,et al.  Virtual and Augmented Reality Applications in Manufacturing , 2004, MIM.

[26]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[28]  Steven van de Par,et al.  A Computationally-Efficient and Perceptually-Plausible Algorithm for Binaural Room Impulse Response Simulation , 2014 .

[29]  Alessandro Soranzo,et al.  The Use of Virtual Reality in Psychology: A Case Study in Visual Perception , 2015, Comput. Math. Methods Medicine.

[30]  Adrian Hilton,et al.  Room Layout Estimation with Object and Material Attributes Information Using a Spherical Camera , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[31]  Ville Pulkki,et al.  Spatial Sound Reproduction with Directional Audio Coding , 2007 .

[32]  Jont B. Allen,et al.  Image method for efficiently simulating small‐room acoustics , 1976 .

[33]  Richard Szeliski,et al.  Reconstructing building interiors from images , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[34]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Edward H. Adelson,et al.  Exploring features in a Bayesian framework for material recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Annika Neidhardt,et al.  Plausibility of an Interactive Approaching Motion towards a Virtual Sound Source Based on Simplified BRIR Sets , 2018 .

[37]  Hideki Koike,et al.  Deep Neural Networks for Cross-Modal Estimations of Acoustic Reverberation Characteristics from Two-Dimensional Images , 2018 .

[38]  Björn Stenger,et al.  Pano2CAD: Room Layout from a Single Panorama Image , 2016, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[39]  Adrian Hilton,et al.  Acoustic Room Modelling using a Spherical Camera for Reverberant Spatial Audio Objects , 2017 .

[40]  J. Deutsch,et al.  Virtual Reality for Stroke Rehabilitation , 2011, The Cochrane database of systematic reviews.

[41]  Kate Saenko,et al.  From Virtual to Reality: Fast Adaptation of Virtual Object Detectors to Real Domains , 2014, BMVC.

[42]  Shi-Min Hu,et al.  3D indoor scene modeling from RGB-D data: a survey , 2015, Computational Visual Media.

[43]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[44]  Mark Sandler,et al.  Perception of Mismatched Auditory Distance—Cinematic VR , 2018 .

[45]  Tapio Lokki,et al.  Spatial Decomposition Method for Room Impulse Responses , 2013 .

[46]  Damian Murphy,et al.  Directional Bias Equalization of First-Order Binaural Ambisonic Rendering , 2018 .

[47]  P. Jackson,et al.  Perceptual Thresholds of Audio-Visual Spatial Coherence for a Variety of Audio-Visual Objects , 2018 .

[48]  Zihou Meng,et al.  The Just Noticeable Difference of Noise Length and Reverberation Perception , 2006, 2006 International Symposium on Communications and Information Technologies.

[49]  Angelo Farina,et al.  Individualized HRTF for Playing VR Videos with Ambisonics Spatial Audio on HMDs , 2018 .

[50]  Michela Ott,et al.  A LITERATURE REVIEW ON IMMERSIVE VIRTUAL REALITY IN EDUCATION: STATE OF THE ART AND PERSPECTIVES. , 2015, 11th International Conference eLearning and Software for Education.

[51]  Richard Szeliski,et al.  Interactive 3D architectural modeling from unordered photo collections , 2008, SIGGRAPH Asia '08.

[52]  Michael Vorlnder,et al.  Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic Virtual Reality , 2020 .

[53]  Stefan Weinzierl,et al.  Assessing the plausibility of virtual acoustic environments , 2012 .

[54]  Tapio Lokki,et al.  Parametric Multidirectional Decomposition of Microphone Recordings for Broadband High-Order Ambisonic Encoding , 2018 .