A framework for computer-assisted sound design systems supported by modelling affective and perceptual properties of soundscape

ABSTRACT Autonomously generating artificial soundscapes for video games, virtual reality, and sound art presents several non-trivial challenges. We outline a system called Audio Metaphor that is built upon the notion that sound design for soundscape compositions is emotionally informed. We first define the problem space of generating soundscape compositions referencing the sound design and soundscape literature. Next, we survey the state-of-the-art soundscape generation systems and establish the characteristics and challenges for evaluating these types of systems. We then describe the Audio Metaphor system that aims to model the soundscape generation problem using a method of soundscape emotion recognition and segmentation based on perceptual classes, and an autonomous mixing engine utilising optimisation and prediction algorithms to generate a soundscape composition. We evaluate the soundscape compositions generated by Audio Metaphor by comparing them with those created by a human expert and also those generated randomly. Our analysis of the evaluation study reveals that the proposed soundscape generation model is human-competitive regarding semantic and emotion-based indicators.

[1]  N. Pearse,et al.  Deciding on the Scale Granularity of Response Categories of Likert Type Scales: The Case of a 21-Point Scale , 2011 .

[2]  David Birchfield,et al.  Design of a Generative Model for Soundscape Creation , 2005, ICMC.

[3]  David Sonnenschein,et al.  Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema , 2001 .

[4]  R. Fay.  Acoustic Communication , 2003, Springer Handbook of Auditory Research.

[5]  Linda Candy,et al.  Supporting the creative user: a criteria-based approach to interaction design , 1997 .

[6]  Stefan Kersten,et al.  Authoring augmented soundscapes with user-contributed content , 2011 .

[7]  Philippe Pasquier,et al.  Impress: A Machine Learning Approach to Soundscape Affect Classification for a Music Performance Environment , 2013, NIME.

[8]  G. A. Mendelsohn,et al.  Affect grid : A single-item scale of pleasure and arousal , 1989 .

[9]  Rebecca Cain,et al.  A positive soundscape evaluation tool , 2009 .

[10]  Athanasia Zlatintsi,et al.  A supervised approach to movie emotion tracking , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Philippe Pasquier,et al.  Computationally Created Soundscapes with Audio Metaphor , 2013, ICCC.

[12]  Philippe Pasquier,et al.  BF-Classifier: Background/Foreground Classification and Segmentation of Soundscape Recordings , 2015, AM '15.

[13]  Steven Minton,et al.  Minimizing Conflicts: A Heuristic Repair Method for Constraint Satisfaction and Scheduling Problems , 1992, Artif. Intell..

[14]  Jordi Janer,et al.  Soundscape Generation for Virtual Environments using Community-Provided Audio Databases , 2010 .

[15]  Arne Eigenfeldt,et al.  Audio Metaphor: Audio Information Retrieval for Soundscape Composition , 2012 .

[16]  Simon Colton,et al.  On impact and evaluation in computational creativity: a discussion of the Turing Test and an alternative proposal , 2011 .

[17]  William J. Davies,et al.  Development of a soundscape simulator tool , 2009 .

[18]  Greg deBeer.  Pro Tools 10 for Game Audio , 2012 .

[19]  Vincenzo Lombardo,et al.  A Framework for Soundscape Analysis And Re-synthesis , 2009 .

[20]  Catherine Lavandier,et al.  Comparison between multiple linear regressions and artificial neural networks to predict urban sound quality , 2010 .

[21]  Jason Freeman,et al.  Soundscape Composition and Field Recording as a Platform for Collaborative Creativity , 2011, Organised Sound.

[22]  Birgitta Berglund,et al.  Soundscape psychophysics in place , 2007 .

[23]  Anna Jordanous.  Evaluating Evaluation: Assessing Progress in Computational Creativity Research , 2011, ICCC.

[24]  K. Kallinen,et al.  Emotion perceived and emotion felt: Same and different , 2006 .

[25]  Etienne B. Roesch,et al.  A Blueprint for Affective Computing: A Sourcebook and Manual , 2010 .

[26]  Andy Farnell,et al.  Designing Sound , 2008 .

[27]  Barry Truax,et al.  Sound, Listening and Place: The aesthetic dilemma , 2012, Organised Sound.

[28]  Xavier Serra,et al.  Freesound Radio: supporting music creation by exploration of a sound database , 2009 .

[29]  Mathieu Lagrange,et al.  SimScene : a web-based acoustic scenes simulator , 2015 .

[30]  Jean-François Augoyard,et al.  Sonic Experience: A Guide to Everyday Sounds , 2006 .

[31]  Christopher Ariza,et al.  The Interrogator as Critic: The Turing Test and the Evaluation of Generative Music Systems , 2009, Computer Music Journal.

[32]  Stefania Serafin,et al.  Sound Design to Enhance Presence in Photorealistic Virtual Reality , 2004, ICAD.

[33]  Simon Fraser,et al.  SOUNDSCAPE EMOTION RECOGNITION VIA DEEP LEARNING , 2018 .

[34]  Celine Latulipe,et al.  Quantifying the Creativity Support of Digital Tools through the Creativity Support Index , 2014, ACM Trans. Comput. Hum. Interact..

[35]  Jordi Janer,et al.  An Online Platform for Interactive Soundscapes with User-Contributed Audio Content , 2011 .

[36]  Xavier Serra,et al.  Ecological Acoustics Perspective for Content-Based Retrieval of Environmental Sounds , 2010, EURASIP J. Audio Speech Music. Process..

[37]  Alexander Brandon.  Audio for Games: Planning, Process, and Production (New Riders Games) , 2004 .

[38]  R. M. Schafer,et al.  The Soundscape: Our Sonic Environment and the Tuning of the World , 1993 .

[39]  Xavier Serra,et al.  Freesound 2: An Improved Platform for Sharing Audio Clips , 2011 .

[40]  M. Boden.  The creative mind : myths & mechanisms , 1991 .

[41]  John E. W. Poxon,et al.  An exploratory evaluation of perceptual, psychoacoustic and acoustical properties of urban soundscapes , 2013 .

[42]  Philippe Pasquier,et al.  Automatic Soundscape Affect Recognition Using A Dimensional Approach , 2016 .

[43]  Philippe Pasquier,et al.  Automatic Recognition of Eventfulness and Pleasantness of Soundscape , 2015, AM '15.

[44]  Arne Eigenfeldt,et al.  A Methodology for the Comparison of Melodic Generation Models Using Meta-Melo , 2013, ISMIR.

[45]  D. Dubois,et al.  In search for soundscape indicators : Physical descriptions of semantic categories , 2006 .

[46]  Jeffrey J. Scott,et al.  MUSIC EMOTION RECOGNITION: A STATE OF THE ART REVIEW , 2010 .

[48]  Mark Shepard.  Tactical sound garden toolkit , 2007, SIGGRAPH '07.

[49]  Arne Eigenfeldt,et al.  Negotiated Content: Generative Soundscape Composition by Autonomous Musical Agents in Coming Together: Freesound , 2011, ICCC.

[50]  J. Gratch,et al.  The Oxford Handbook of Affective Computing , 2014 .

[51]  William W. Gaver.  What in the World Do We Hear? An Ecological Approach to Auditory Event Perception , 1993 .

[52]  Paul Lamere,et al.  Social Tagging and Music Information Retrieval , 2008 .

[53]  Barry Truax,et al.  Soundscape, Acoustic Communication and Environmental Sound Composition , 1996 .

[54]  R. Garland.  The Mid-Point on a Rating Scale: Is it Desirable? , 1991 .

[55]  William J. Davies,et al.  The positive soundscape project , 2007 .

[56]  P. Symonds,et al.  On the Loss of Reliability in Ratings Due to Coarseness of the Scale. , 1924 .

[57]  J. Jacoby,et al.  Is There an Optimal Number of Alternatives for Likert Scale Items? Study I: Reliability and Validity , 1971 .

[58]  Dan Ventura.  A Reductio Ad Absurdum Experiment in Sufficiency for Evaluating (Computational) Creative Systems , 2008 .

[59]  Justin Salamon,et al.  Scaper: A library for soundscape synthesis and augmentation , 2017, 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA).

[60]  Bruce Thompson,et al.  Score Reliability in Webor Internet-Based Surveys: Unnumbered Graphic Rating Scales versus Likert-Type Scales , 2001 .

[61]  Philippe Pasquier,et al.  Soundscape Audio Signal Classification and Segmentation Using Listeners Perception of Background and Foreground Sound , 2016 .

[62]  Danièle Dubois,et al.  CATEGORIES FOR SOUNDSCAPE : TOWARD A HYBRID CLASSIFICATION , 2010 .

[63]  Philippe Pasquier,et al.  Emo-soundscapes: A dataset for soundscape emotion recognition , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[64]  Dick Botteldooren,et al.  The temporal structure of urban soundscapes , 2006 .

[65]  Andrea Valle,et al.  Imaginary soundscapes: the SoDA project , 2014, Audio Mostly Conference.

[66]  Sergio L. Toral Marín,et al.  Small world networks and creativity in audio clip sharing , 2012, Int. J. Soc. Netw. Min..

[67]  Geraint A. Wiggins.  Searching for computational creativity , 2006, New Generation Computing.

[68]  Geraint A. Wiggins,et al.  Towards A Framework for the Evaluation of Machine Compositions , 2001 .

[69]  Arthur Flexer,et al.  Statistical evaluation of music information retrieval experiments , 2006 .