Synchronization of Audio-Visual Elements in Web Applications

The web offers rich visual content, and for many web designers it is the visual domain that dominates their practice. Yet sound, music and audio all play a significant role in the human sensory systems, in some cases sound can be more powerful, especially to emotionally influence viewers. This paper looks at the relationship between audio and visual elements in web content. In particular looking at the power of synchronization of audio and visual content. Bringing together the sensory experience from the two sensory systems to potentially enhance the users overall experience. The relationship between audio and visual elements, and more importantly their level of synchronization was explored through an experimental trial where users were exposed to web content with differing levels of synchronization between the audio and visual components. User feedback from these trials showed that synchronization played a key role in the content that users selected as their preferred items. From these results, several new principles for the effective design of web based audio-visual elements were developed including linking the nature and complexity of the visual forms, to the nature and complexity of the audio forms, to provide greater synchronization and enhance overall user experience.

[1]  Dale Patterson,et al.  3D SPACE: using depth and movement for selection tasks , 2007, Web3D '07.

[2]  Teresa Chambel,et al.  Synesthetic video: hearing colors, seeing sounds , 2010, MindTrek.

[3]  Kathrin Fahlenbrach.  Aesthetics and Audiovisual Metaphors in Media Perception , 2005 .

[4]  B. Snyder.  Music and Memory: An Introduction , 2001 .

[5]  Simon Baron-Cohen,et al.  Synaesthesia : classic and contemporary readings , 1997 .

[6]  Bo Jiang,et al.  Brain computer interface based 3D game for attention training and rehabilitation , 2011, 2011 6th IEEE Conference on Industrial Electronics and Applications.

[7]  Dale Patterson,et al.  3D Space: Special Project in Advanced Computer Environments , 2003 .

[8]  Jon Driver,et al.  Sight and sound out of synch: Fragmentation and renormalisation of audiovisual integration and subjective timing , 2013, Cortex.

[9]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .

[10]  John Whitney,et al.  Digital Harmony: On the Complementarity of Music and Visual Art , 1980 .

[11]  Richard E. Cytowic.  Synesthesia: A union of the senses, 2nd ed. , 2002 .

[12]  Tsuhan Chen,et al.  Audio-visual integration in multimodal communication , 1998, Proc. IEEE.

[13]  Michel Chion,et al.  Audio-Vision: Sound on Screen , 1994 .

[14]  Mitchell Whitelaw,et al.  Synesthesia and Cross-Modality in Contemporary Audiovisuals , 2008 .

[15]  Chia-Jung Tsay.  Sight over sound in the judgment of music performance , 2013, Proceedings of the National Academy of Sciences.

[16]  D. Graber.  Say it with Pictures , 1996 .

[17]  Dale Patterson.  Using Interactive 3D Game Play to Make Complex Medical Knowledge More Accessible , 2014, ICCS.

[18]  Carol Vernallis.  Experiencing Music Video: Aesthetics and Cultural Context , 2004 .

[19]  Q. Summerfield,et al.  Lipreading and audio-visual speech perception. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.