Participatory Design of Sonification Development for Learning about Molecular Structures in Virtual Reality

Background: Chemistry and biology students often have difficulty understanding molecular structures. Sonification (the rendition of data into non-speech sounds that convey information) can be used to support molecular understanding by complementing scientific visualization. A proper sonification design is important for its effective educational use. This paper describes a participatory design (PD) approach to designing and developing the sonification of a molecular structure model to be used in an educational setting. Methods: Biology, music, and computer science students and specialists designed a sonification of a model of an insulin molecule, following Spinuzzi’s PD methodology and involving evolutionary prototyping. The sonification was developed using open-source software tools used in digital music composition. Results and Conclusions: We tested our sonification played on a virtual reality headset with 15 computer science students. Questionnaire and observational results showed that multidisciplinary PD was useful and effective for developing an educational scientific sonification. PD allowed for speeding up and improving our sonification design and development. Making a usable (effective, efficient, and pleasant to use) sonification of molecular information requires the multidisciplinary participation of people with music, computer science, and molecular biology backgrounds.

[1]  L. Tinoca,et al.  Design-Based Research in the Educational Field: A Systematic Literature Review , 2022, Education Sciences.

[2]  Pei-Ying Chen,et al.  Three decades of game-based learning in science and mathematics education: an integrated bibliometric analysis and systematic review , 2021, Journal of Computers in Education.

[3]  J. Bailenson,et al.  Participatory research on using virtual reality to teach ocean acidification: a study in the marine education community , 2020, Environmental Education Research.

[4]  David R. Glowacki,et al.  Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and carbon nanotubes , 2020, Audio Mostly Conference.

[5]  M. Buehler,et al.  Sonification based de novo protein design using artificial intelligence, structure prediction, and analysis using molecular modeling , 2020, APL bioengineering.

[6]  Tifanie Bouchara,et al.  Immersive sonification of protein surface , 2018, 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).

[7]  Ziyang Li,et al.  The Application of Student Participation in the Design of Virtual Reality Educational Products , 2019, HCI.

[8]  Pedro C. Santana-Mancilla,et al.  A Usability Study of an Interactive Auditory Display for Supporting Learning of Molecular Structure , 2019 .

[9]  K. Vines,et al.  Sonification of numerical data for education , 2019, Open Learning: The Journal of Open, Distance and e-Learning.

[10]  José L. Arciniegas,et al.  Participatory design method: Co-Creating user interfaces for an educational interactive system , 2018, Interacción.

[11]  Morgan Watson,et al.  Student difficulties during structure–function problem solving , 2018, Biochemistry and molecular biology education : a bimonthly publication of the International Union of Biochemistry and Molecular Biology.

[12]  David Benyon,et al.  Investigating Effective Methods of Designing Sonifications , 2018, Proceedings of the 24th International Conference on Auditory Display - ICAD 2018.

[13]  Cumhur Erkut,et al.  Sonic Interactions in Virtual Reality: State of the Art, Current Challenges, and Future Directions , 2018, IEEE Computer Graphics and Applications.

[14]  James R. Lewis,et al.  The System Usability Scale: Past, Present, and Future , 2018, Int. J. Hum. Comput. Interact..

[15]  Myounghoon Jeon,et al.  Participatory Design Research Methodologies: A Case Study in Dancer Sonification , 2017 .

[16]  Euan Freeman,et al.  Multimodal feedback in HCI: haptics, non-speech audio, and their applications , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[17]  Mark D. Temple,et al.  An auditory display tool for DNA sequence analysis , 2017, BMC Bioinformatics.

[18]  Shigeki Hirobayashi,et al.  Creating an Adaptive Technology Using a Cheminformatics System To Read Aloud Chemical Compound Names for People with Visual Disabilities , 2016 .

[19]  Dan Gärdenfors,et al.  Auditory Interfaces , 2016 .

[20]  Oussama Metatla,et al.  Sonifications for digital audio workstations: Reflections on a participatory design approach , 2015, ICAD.

[21]  Robert Höldrich,et al.  Observations on an interdisciplinary design process using a sonification framework , 2015, ICAD.

[22]  Thomas Ertl,et al.  Enhancing visualization of molecular simulations using sonification , 2015, 2015 IEEE 1st International Workshop on Virtual and Augmented Reality for Molecular Science (VARMS@IEEEVR).

[23]  Antonella De Angeli,et al.  Applying user centred and participatory design approaches to commercial product development , 2014 .

[24]  Federico Visi,et al.  UNFOLDING | CLUSTERS : A MUSIC AND VISUAL MEDIA MODEL OF ALS PATHOPHYSIOLOGY , 2014 .

[25]  Jesper Simonsen,et al.  Participatory Design: an introduction , 2012 .

[26]  Huseyin Seker,et al.  The sound of proteins , 2012, Proceedings of 2012 IEEE-EMBS International Conference on Biomedical and Health Informatics.

[27]  Kathleen M. MacQueen,et al.  Applied Thematic Analysis , 2011 .

[28]  M. Towns,et al.  A Review of Spatial Ability Literature, Its Connection to Chemistry, and Implications for Instruction , 2011 .

[29]  Davide Rocchesso,et al.  The Sonification Handbook , 2011 .

[30]  Henrik Kniberg,et al.  Kanban and Scrum - Making the Most of Both , 2010 .

[31]  Thomas Hermann,et al.  BROWSING RNA STRUCTURES BY INTERACTIVE SONIFICATION , 2010 .

[32]  Mehdi Ammi,et al.  Multisensory VR interaction for protein-docking in the CoRSAIRe project , 2009, Virtual Reality.

[33]  Dana Chisnell,et al.  Handbook of Usability Testing , 2009 .

[34]  Rahul Sawhney,et al.  Sonic Grid: an auditory interface for the visually impaired to navigate GUI-based environments , 2008, IUI '08.

[35]  Arthur Edwards Block,et al.  Auditory Display as a Tool for Teaching Network Intrusion Detection , 2007, Int. J. Emerg. Technol. Learn..

[36]  Miguel Ángel García-Ruíz,et al.  An overview of auditory display to assist comprehension of molecular information , 2006, Interact. Comput..

[37]  Simon Fraser,et al.  The Role of Participatory Workshops in Investigating Narrative and Sound Ecologies in the Design of an Ambient Intelligence Audio Display , 2006 .

[38]  Gavin Sim,et al.  Assessing usability and fun in educational software , 2005, IDC '05.

[39]  Clay Spinuzzi,et al.  The Methodology of Participatory Design , 2005 .

[40]  Andy Hunt,et al.  Guest Editors' Introduction: An Introduction to Interactive Sonification , 2005, IEEE Multim..

[41]  Jeffrey R. Appling,et al.  Instructional Technology and Molecular Visualization , 2004 .

[42]  Haluk Özmen,et al.  Some Student Misconceptions in Chemistry: A Literature Review of Chemical Bonding , 2004 .

[43]  Sari Kujala,et al.  User involvement: A review of the benefits and challenges , 2003, Behav. Inf. Technol..

[44]  Robert Upson,et al.  Educational sonification exercises: Pathways for mathematics and musical achievement , 2002 .

[45]  Elizabeth B.-N. Sanders,et al.  From user-centered to participatory design approaches , 2002 .

[46]  Miguel A. Garcia-Ruiz,et al.  Binding Virtual Molecules Sounds Good!: Exploring a Novel Way to Convey Molecular Bonding to Students. , 2002 .

[47]  Michael J. Muller,et al.  Participatory design: the third space in HCI , 2002 .

[48]  Stephen Brewster,et al.  Experimentally Derived Guidelines for the Creation of Earcons , 2001 .

[49]  Mary Anne Clark,et al.  Life Music: The Sonification of Proteins , 1999, Leonardo.

[50]  Alex T. Pang,et al.  PROMUSE: A System for Multi-Media Data Presentation of Protein Structural Alignments , 1998, Pacific Symposium on Biocomputing.

[51]  James P. Birk,et al.  Effect of Experience on Retention and Elimination of Misconceptions about Molecular Structure and Bonding , 1999 .

[52]  John G. Neuhoff,et al.  Sonification Report: Status of the Field and Research Agenda Prepared for the National Science Foundation by members of the International Community for Auditory Display , 1999 .

[53]  Patrick W. Jordan,et al.  An Introduction to Usability , 1998 .

[54]  Theresa Julia Zielinski,et al.  Using computers in chemistry and chemical education , 1997 .

[55]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[56]  P. Y. Bruice.  Essential Organic Chemistry , 1995 .

[57]  Gregory Kramer,et al.  An introduction to auditory display , 1994 .

[58]  Meera M. Blattner,et al.  Communicating and Learning Through Non-speech Audio , 1992 .

[59]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[60]  N. Alcock.  Bonding and structure : structural principles in inorganic and organic chemistry , 1990 .

[61]  W. Doll,et al.  A discrepancy model of end-user computing involvement , 1989 .

[62]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..

[63]  Blake Ives,et al.  An empirical study of the impact of user involvement on system usage and information satisfaction , 1986, CACM.

[64]  Quentin R. Petersen,et al.  Some Reflections on the Use and Abuse of Molecular Models. , 1970 .