Enhancing visualization of molecular simulations using sonification

Scientific visualization is an application area for virtual reality environments like stereoscopic displays or CAVEs. Especially interactive molecular visualizations that show the complex three-dimensional structures found in structural biology are often investigated using such environments. In contrast to VR applications like simulators, molecular visualization typically lacks auditory output. Nevertheless, sonification can be used to convey information about the data. In our work, we use sound to highlight events extracted from a molecular dynamics simulation. This not only offloads information from the visual channel, but can also guide the attention of the analyst towards important phenomena even if they are occluded in the visualization. Sound also creates a higher level of immersion, which can be beneficial for educational purposes. In this paper, we detail our application that adds sonification to the visualization of molecular simulations.

[1]  Sook Young Won.  Auditory display of genome data: Human chromosome 21 , 2005 .

[2]  D. Goodsell,et al.  Visualization of macromolecular structures , 2010, Nature Methods.

[3]  Gregory Berenato,et al.  A Simple Audio Conductivity Device , 1997 .

[4]  John G. Neuhoff,et al.  Sonification Report: Status of the Field and Research Agenda Prepared for the National Science Foundation by members of the International Community for Auditory Display , 1999 .

[5]  R. D. de Haan,et al.  The eye of the beholder: inter-rater agreement among experts on psychogenic jerky movement disorders , 2013, Journal of Neurology, Neurosurgery & Psychiatry.

[6]  Thomas Ertl,et al.  MegaMol—A Prototyping Framework for Particle-Based Visualization , 2015, IEEE Transactions on Visualization and Computer Graphics.

[7]  Inger Ekman.  On the desire to not kill your players: Rethinking sound in pervasive and mixed reality games , 2013, FDG.

[8]  Dan Gärdenfors,et al.  Audio games: new perspectives on game audio , 2004, ACE '04.

[9]  Edward S. Yeung,et al.  Pattern recognition by audio representation of multivariate analytical data , 1980 .

[10]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .

[11]  Terri L. Bonebright,et al.  Sonific ation Report: Status of the Field and Research Agenda , 2010 .

[12]  Thomas Hermann,et al.  Sonification for Exploratory Data Analysis , 2002 .

[13]  Thierry Delatour.  Molecular Music: The Acoustic Conversion of Molecular Vibrational Spectra , 2000, Computer Music Journal.

[14]  Brian F. G. Katz,et al.  Non-visual identification, localization, and selection of entities of interest in a 3D environment , 2014, Journal on Multimodal User Interfaces.

[15]  Susan H. Hixson,et al.  Development of a Data Acquisition and Data Analysis System for Visually Impaired Chemistry Students , 1994 .

[16]  Mehdi Ammi,et al.  Multisensory VR interaction for protein-docking in the CoRSAIRe project , 2009, Virtual Reality.

[17]  Alex T. Pang,et al.  PROMUSE: A System for Multi-Media Data Presentation of Protein Structural Alignments , 1998, Pacific Symposium on Biocomputing.

[18]  Antoine Ferreira,et al.  Prototyping bio-nanorobots using molecular dynamics simulation and virtual reality , 2008, Microelectron. J..

[19]  Thomas Ertl,et al.  GPU-powered tools boost molecular visualization , 2011, Briefings Bioinform..

[20]  Falko Kuester,et al.  The StarCAVE, a third-generation CAVE and virtual reality OptIPortal , 2009, Future Gener. Comput. Syst..

[21]  Miguel Ángel García-Ruíz,et al.  An overview of auditory display to assist comprehension of molecular information , 2006, Interact. Comput..

[22]  Simon McIntosh-Smith,et al.  A GPU-accelerated immersive audio-visual framework for interaction with molecular dynamics using consumer depth sensors. , 2014, Faraday discussions.

[23]  Tassos A. Mikropoulos,et al.  Educational virtual environments: A ten-year review of empirical research (1999-2009) , 2011, Comput. Educ..

[24]  John F. Veiga,et al.  The Eye of the Beholder Revisited , 1976 .

[25]  H. Geiger,et al.  Elektronenzählrohr zur Messung schwächster Aktivitäten , 1928, Naturwissenschaften.

[26]  Ross D. King,et al.  PM - protein music , 1996, Comput. Appl. Biosci..

[27]  Thomas Ertl,et al.  Parallel Contour-Buildup algorithm for the molecular surface , 2011, 2011 IEEE Symposium on Biological Data Visualization (BioVis)..

[28]  Stefan Birmanns,et al.  GPU-accelerated visualization of protein dynamics in ribbon mode , 2011, Electronic Imaging.

[29]  Sara Bly,et al.  Presenting information in sound , 1982, CHI '82.

[30]  Jonathan C. Roberts,et al.  Visualization for the Physical Sciences , 2012, Comput. Graph. Forum.

[31]  Huseyin Seker,et al.  The sound of proteins , 2012, Proceedings of 2012 IEEE-EMBS International Conference on Biomedical and Health Informatics.

[32]  Mark R. Anderson,et al.  Direct comparison of the impact of head tracking, reverberation, and individualized head-related transfer functions on the spatial perception of a virtual speech source. , 2001, Journal of the Audio Engineering Society. Audio Engineering Society.

[33]  John C. Hart,et al.  The CAVE: audio visual experience automatic virtual environment , 1992, CACM.

[34]  Perry R. Cook,et al.  Synthesis toolkit in C , 2002, SIGGRAPH 1996.

[35]  S. Carlile,et al.  Measuring the human head-related transfer functions: a novel method for the construction and calibration of a miniature "in-ear" recording system. , 1994, The Journal of the Acoustical Society of America.

[36]  Mary Anne Clark,et al.  Life Music: The Sonification of Proteins , 1999, Leonardo.

[37]  Richard May,et al.  A Survey of Large High-Resolution Display Technologies, Techniques, and Applications , 2006, IEEE Virtual Reality Conference (VR 2006).

[38]  Helge Ritter,et al.  Listen to your Data: Model-Based Sonification for Data Analysis , 1999 .

[39]  KENSHI HAYASHI,et al.  Basically musical , 1984, Nature.

[40]  Woon-Seng Gan,et al.  User-defined spectral manipulation of HRTF for improved localisation in 3D sound systems , 1998 .

[41]  Florian Grand,et al.  Sumo. A Sonification Utility for Molecules , 2008 .

[42]  Larry S. Davis,et al.  Virtual audio system customization using visual matching of ear parameters , 2002, Object recognition supported by user interaction for service robots.

[43]  D. Klionsky,et al.  SNARE Dance , 2012, Autophagy.

[44]  Flemming Christensen,et al.  The importance of head movements for binaural room synthesis - a pilot experiment , 2000 .

[45]  Tim Weyrich,et al.  Eurographics Symposium on Point-based Graphics (2006) Gpu-based Ray-casting of Quadratic Surfaces , 2022 .

[46]  Matthieu Chavent,et al.  Advances in Human-Protein Interaction - Interactive and Immersive Molecular Simulations , 2012 .