Sketching sounds: an exploratory study on sound-shape associations

Sound synthesiser controls typically correspond to technical parameters of signal processing algorithms rather than intuitive sound descriptors that relate to human perception of sound. This makes it difficult to realise sound ideas in a straightforward way. Cross-modal mappings, for example between gestures and sound, have been suggested as a more intuitive control mechanism. A large body of research shows consistency in human associations between sounds and shapes. However, the use of drawings to drive sound synthesis has not been explored to its full extent. This paper presents an exploratory study that asked participants to sketch visual imagery of sounds with a monochromatic digital drawing interface, with the aim to identify different representational approaches and determine whether timbral sound characteristics can be communicated reliably through visual sketches. Results imply that the development of a synthesiser exploiting sound-shape associations is feasible, but a larger and more focused dataset is needed in followup studies.

[1]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[2]  Manuel J. Fonseca,et al.  Sketch-based retrieval of drawings using spatial proximity , 2010, J. Vis. Lang. Comput..

[3]  Peter Knees,et al.  Searching for Audio by Sketching Mental Images of Sound: A Brave New Idea for Audio Retrieval in Creative Music Production , 2016, ICMR.

[4]  Arthur Flexer,et al.  Visualization of perceptual qualities in Textural sounds , 2012, ICMC.

[5]  S. Ystad,et al.  Timbre semantics through the lens of crossmodal correspondences: A new way of asking old questions , 2018, Acoustical Science and Technology.

[6]  Lars Engeln,et al.  CoHEARence of audible shapes—a qualitative user study for coherent visual audio design with resynthesized shapes , 2020, Personal and Ubiquitous Computing.

[7]  D. Rendall,et al.  Parsing the role of consonants versus vowels in the classic Takete-Maluma phenomenon. , 2013, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[8]  Brian Eoff,et al.  ShortStraw: a simple and effective corner finder for polylines , 2008, SBM'08.

[9]  Yuting Zhang,et al.  Sketch-Based Image Retrieval by Salient Contour Reinforcement , 2016, IEEE Transactions on Multimedia.

[10]  Stephen McAdams,et al.  The Perception of Musical Timbre , 2008 .

[11]  Philippe Esling,et al.  FlowSynth: Simplifying Complex Audio Generation Through Explorable Latent Spaces with Normalizing Flows , 2020, IJCAI.

[12]  M. M. Taylor,et al.  Phonetic symbolism in four unrelated languages. , 1962, Canadian journal of psychology.

[13]  R. Davis.  The fitness of names to drawings. A cross-cultural study in Tanganyika. , 1961, British journal of psychology.

[14]  David K. Blake.  Timbre as Differentiation in Indie Music , 2012 .

[15]  D. Maurer,et al.  The shape of boubas: sound-shape correspondences in toddlers and adults. , 2006, Developmental science.

[16]  Kostas Giannakis,et al.  A comparative evaluation of auditory-visual mappings for sound visualisation , 2006, Organised Sound.

[17]  Daniel Müllensiefen,et al.  The Musicality of Non-Musicians: An Index for Assessing Musical Sophistication in the General Population , 2014, PloS one.

[18]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal of chiropractic medicine.

[19]  M. Thaut,et al.  The Oxford handbook of music psychology , 2011 .

[20]  V. Ramachandran,et al.  Synaesthesia? A window into perception, thought and language , 2001 .

[21]  Richard Kronland-Martinet,et al.  Controlling the Perceived Material in an Impact Sound Synthesizer , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[22]  Dan Tidhar,et al.  Musicians are more consistent: Gestural cross-modal mappings of pitch, loudness and tempo in real-time , 2014, Front. Psychol..

[23]  Stephen McAdams,et al.  A Comparison of Approaches to Timbre Descriptors in Music Information Retrieval and Music Psychology , 2016 .

[24]  W. Köhler.  Gestalt psychology , 1967 .

[25]  Why Can You Hear a Difference between Pouring Hot and Cold Water? An Investigation of Temperature Dependence in Psychoacoustics , 2018 .

[26]  Perry R. Cook,et al.  The Featsynth Framework for Feature-Based synthesis: Design and Applications , 2007, ICMC.

[27]  Jean Rouat,et al.  Audiovisual correspondence between musical timbre and visual shapes , 2014, Front. Hum. Neurosci..

[28]  Jack Bresenham,et al.  Algorithm for computer control of a digital plotter , 1965, IBM Syst. J..

[29]  Lawrence E. Marks,et al.  Synesthesia: Strong and Weak , 2001 .

[30]  Tim Brookes,et al.  Modelling Timbral Hardness , 2019, Applied Sciences.

[31]  Joseph J. LaViola,et al.  Revisiting ShortStraw: improving corner finding in sketch-based interfaces , 2009, SBIM '09.

[32]  Inger Ekman,et al.  Using vocal sketching for designing sonic interactions , 2010, Conference on Designing Interactive Systems.

[33]  C. Spence,et al.  “Bouba” and “Kiki” in Namibia? A remote culture make similar shape–sound matches, but different shape–taste matches to Westerners , 2013, Cognition.

[34]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[35]  Etienne Richan,et al.  A proposal and evaluation of new timbre visualization methods for audio sample browsers , 2020, Personal and Ubiquitous Computing.

[36]  O. Collignon,et al.  Sound symbolism in sighted and blind. The role of vision and orthography in sound-shape correspondences , 2019, Cognition.