Game Sound Technology and Player Interaction: Concepts and Developments

This chapter treats computer game playing as an affective activity, largely guided by the audio-visual aesthetics of game content (of which, here, we concentrate on the role of sound) and the pleasure of gameplay. To understand the aesthetic impact of game sound on player experience, definitions of emotions are briefly discussed and framed in the game context. This leads to an introduction of empirical methods for assessing physiological and psychological effects of play, such as the affective impact of sonic player-game interaction. The psychological methodology presented is largely based on subjective interpretation of experience, while psychophysiological methodology is based on measurable bodily changes, such as context-dependent, physiological experience. As a means to illustrate both the potential and the difficulties inherent in such methodology we discuss the results of some experiments that investigate game sound and music effects and, finally, we close with a discussion of possible research directions based on a speculative assessment of the future of player-game interaction through affective sound.

[1]  Dan E. Mayers Resident Evil (Motion picture) , 2002 .

[2]  Robert A Jacobs,et al.  Bayesian integration of visual and auditory signals for spatial localization. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Jan Berg,et al.  Digiwall: an interactive climbing wall , 2005, ACE '05.

[4]  R. Wise Dopamine, learning and motivation , 2004, Nature Reviews Neuroscience.

[5]  David Poeppel,et al.  Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony , 2004, Speech Commun..

[6]  Jeff T. Larsen,et al.  The psychophysiology of emotion. , 1993 .

[7]  Konrad P. Körding,et al.  Sparse Spectrotemporal Coding of Sounds , 2003, EURASIP J. Adv. Signal Process..

[8]  L. Nadel,et al.  Cognitive neuroscience of emotion , 2000 .

[9]  William W. Gaver What in the World Do We Hear? An Ecological Approach to Auditory Event Perception , 1993 .

[10]  David Hanson Exploring the Aesthetic Range for Humanoid Robots , 2006 .

[11]  Michel Chion,et al.  The Voice in Cinema , 1999 .

[12]  W. Murch Sound Design: The Dancing Shadow , 1995 .

[13]  K. Brodmann Vergleichende Lokalisationslehre der Großhirnrinde : in ihren Prinzipien dargestellt auf Grund des Zellenbaues , 1985 .

[14]  John G. Neuhoff,et al.  Sonification Report: Status of the Field and Research Agenda Prepared for the National Science Foundation by members of the International Community for Auditory Display , 1999 .

[15]  Rec. ITU-R BT.1359-1 1 RECOMMENDATION ITU-R BT.1359-1 RELATIVE TIMING OF SOUND AND VISION FOR BROADCASTING , 1998 .

[16]  Hanns Eisler,et al.  Composing for the Films , 1947 .

[17]  Jean-François Augoyard,et al.  Sonic Experience: A Guide to Everyday Sounds , 2006 .

[18]  Perry R. Cook,et al.  Music, cognition, and computerized sound: an introduction to psychoacoustics , 1999 .

[19]  E Tulving,et al.  Identification of simultaneously presented simple visual and auditory stimuli. , 1967, Acta psychologica.

[20]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[21]  Espen Aarseth Playing Research: Methodological approaches to game analysis , 2003 .

[22]  P. Gouk Raising Spirits and Restoring Souls: Early Modern Medical Explanations for Music's Effects , 2020, Hearing Cultures.

[23]  Matthew Lombard,et al.  At the Heart of It All: The Concept of Presence , 2006 .

[24]  E. Hudlicka AFFECTIVE COMPUTING FOR GAME DESIGN , 2008 .

[25]  Michel Chion,et al.  Audio-Vision: Sound on Screen , 1994 .

[26]  Günter Rudolph,et al.  Evolutionary composition of music with learned melody evaluation , 2006 .

[27]  Ulf Wilhelmsson,et al.  A Combined Model for the Structuring of Computer Game Audio , 2011 .

[28]  Karen Collins From Pac-Man to pop music : interactive audio in games and new media , 2008 .

[29]  B. Kushner Descartes' error. , 1998, Journal of AAPOS : the official publication of the American Association for Pediatric Ophthalmology and Strabismus.

[30]  J. Russell,et al.  The neurophysiological bases of emotion: An fMRI study of the affective circumplex using emotion‐denoting words , 2009, Human brain mapping.

[31]  Ulrich Reiter,et al.  Influence of Interaction on Perceived Quality in Audio Visual Applications: Subjective Assessment with n-Back Working Memory Task, II , 2007 .

[32]  Mark Grimshaw,et al.  The Acoustic Ecology of the First-Person Shooter: The Player Experience of Sound in the First-Person Shooter Computer Game , 2008 .

[33]  Dave Moffat,et al.  Personality Parameters and Programs , 1997, Creating Personalities for Synthetic Actors.

[34]  Gerald Moore,et al.  Rhythmanalysis: Space, Time and Everyday Life , 2005 .

[35]  Shlomo Dubnov,et al.  The topos of music: geometric logic of concepts, theory, and performance , 2005 .

[36]  David Sonnenschein,et al.  Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema , 2001 .

[37]  K. Hugdahl,et al.  The effects of different intensity levels of background noise on dichotic listening to consonant-vowel syllables. , 2008, Scandinavian journal of psychology.

[38]  William M. Hartmann,et al.  Psychoacoustics: Facts and Models , 2001 .

[39]  Mark Grimshaw,et al.  The audio Uncanny Valley: Sound, fear and the horror game , 2009 .

[40]  Gerhard Widmer,et al.  EXPRESSIVE PERFORMANCE RENDERING: INTRODUCING PERFORMANCE CONTEXT , 2009 .

[41]  G. Simmel The Metropolis and Mental Life , 2012 .

[42]  Daniel J. Levitin,et al.  The rewards of music listening: Response and physiological connectivity of the mesolimbic system , 2005, NeuroImage.

[43]  Kirsten Cater,et al.  Parallel worlds: immersion in location-based experiences , 2005, CHI Extended Abstracts.

[44]  P. Manning,et al.  Erving Goffman and Modern Sociology , 1994 .

[45]  F. Alton Everest,et al.  Master handbook of acoustics , 1981 .

[46]  Maic Masuch,et al.  RAY ACOUSTICS USING COMPUTER GRAPHICS TECHNOLOGY , 2007 .

[47]  E. Casey Wardynski,et al.  Introducing Emotion into Military Simulation and Video Game Design America's Army: Operations and VIRTE. , 2002 .

[48]  Terri L. Bonebright,et al.  Sonific ation Report: Status of the Field and Research Agenda , 2010 .

[49]  H. Ishiguro,et al.  The uncanny advantage of using androids in cognitive and social science research , 2006 .

[50]  D. Frisby Cityscapes of Modernity: Critical Explorations , 2001 .

[51]  William A. Yost,et al.  Spatial hearing: The psychophysics of human sound localization, revised edition , 1998 .

[52]  E. Bigand,et al.  Sound to Sense , Sense to Sound A State of the Art in Sound and Music Computing , 2008 .

[53]  Sean A. Spence,et al.  Descartes' Error: Emotion, Reason and the Human Brain , 1995 .

[54]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[55]  Matti Paavola,et al.  3D Audio for Mobile Devices via Java , 2005 .

[56]  B. Perron Sign of a Threat : The Effects of Warning Systems in Survival Horror Games , 2004 .

[57]  A. Vendrik,et al.  CAN A SENSORY SYSTEM BE SPECIFIED BY ITS INTERNAL NOISE? , 1965, The Journal of the Acoustical Society of America.

[58]  R. Altman,et al.  Sound theory, sound practice , 1992 .

[59]  R. Caillois,et al.  Man, Play and Games , 1958 .

[60]  Victor Kaptelinin,et al.  Acting with technology: Activity theory and interaction design , 2006, First Monday.

[61]  B. Langer,et al.  The Practice of Everyday Life , 2019, Forms of Thinking in Leopardi’s Zibaldone.

[62]  S. Puttonen,et al.  The psychophysiology of James Bond: phasic emotional responses to violent video game events. , 2008, Emotion.

[63]  Anne Sullivan,et al.  Auditory perception. , 1973, British medical journal.

[64]  E. Wenzel The impact of system latency on dynamic performance in virtual acoustic environments , 1998 .

[65]  Ulrich Dr.-Ing. Reiter Bimodal Audiovisual Perception in Interactive Application Systems of Moderate Complexity , 2009 .

[66]  Sebastian Aav Adaptive Music System for DirectSound , 2005 .

[67]  K. M. Lee,et al.  1---------------------- Effects of Networked Interactivity in Educational Games-----------------: Mediating Effects of Social Presence , 2007 .

[68]  Artur Sepp,et al.  Filling the Gaps , 1962, Mental health.

[69]  R. Plutchik The Nature of Emotions , 2001 .

[70]  Rosalind W. Picard Affective Computing , 1997 .

[71]  Maic Masuch,et al.  Leaving the screen New perspectives in audio-only gaming , 2005 .

[72]  Jun'ichiro Seyama,et al.  The Uncanny Valley: Effect of Realism on the Impression of Artificial Human Faces , 2007, PRESENCE: Teleoperators and Virtual Environments.

[73]  Axel Berndt Musical Nonlinearity in Interactive Narrative Environments , 2009, ICMC.

[74]  Stephen Brewster,et al.  Providing a Structured Method for Integrating Non-Speech Audio into Human-Computer Interfaces , 1994 .

[75]  Lars Stockmann,et al.  A Musical Instrument based on Interactive Sonification Techniques , 2008 .

[76]  Steve Benford,et al.  Ambiguity as a resource for design , 2003, CHI '03.

[77]  C. Holden Why is this man smiling? , 1989, Science.

[78]  J. Sloboda,et al.  Music and emotion: Theory and research , 2001 .

[79]  John G. Beerends,et al.  The Influence of Video Quality on Perceived Audio Quality and Vice Versa , 1999 .

[80]  Neville H Fletcher,et al.  Principles of Vibration and Sound, 2nd edition , 2004 .

[81]  C. Spence,et al.  The cost of expecting events in the wrong sensory modality , 2001, Perception & psychophysics.

[82]  translated by Margaret Schwartz,et al.  Techno-Cultural Convergence: Wanting to Say Everything, Wanting to Watch Everything , 2009 .

[83]  Keith S. Jones,et al.  The human-web interaction cycle: a proposed and tested framework of perception, cognition, and action on the web , 2003 .

[84]  Stuart Cunningham,et al.  Emotion, Content & Context in Sound and Music , 2010 .

[85]  Sound studio construction on a budget , 1996 .

[86]  Niklas Röber,et al.  Interaction with sound: explorations beyond the frontiers of 3D virtual auditory environments , 2009 .

[87]  M. Bradley,et al.  Affective reactions to acoustic stimuli. , 2000, Psychophysiology.

[88]  Mel Slater,et al.  Presence and The Sixth Sense , 2002, Presence: Teleoperators & Virtual Environments.

[89]  The Digital Eye : Image Metrics Attempts to Leap the Uncanny Valley , 2011 .

[90]  C. Hong Dance Dance Revolution , 2007 .

[91]  Hanna Wirman,et al.  Extending experiences : structure, analysis and design of computer game player experience , 2008 .

[92]  J. Blauert Spatial Hearing: The Psychophysics of Human Sound Localization , 1983 .

[93]  D. Watson,et al.  Toward a consensual structure of mood. , 1985, Psychological bulletin.

[94]  Eoin Mullan Physical Modelling for Sound Synthesis , 2011 .

[95]  Chris Cornelis,et al.  Valued Constraint Satisfaction Problems Applied to Functional Harmony , 2009, IFSA/EUSFLAT Conf..

[96]  Kristine Jørgensen Time for New Terminology? Diegetic and Non-Diegetic Sounds in Computer Games Revisited , 2010 .

[97]  P. Lang The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[98]  Guillermo Pozzati,et al.  Infinite Suite: Computers and Musical Form , 2009, ICMC.

[99]  Jonas Braasch,et al.  Modelling of Binaural Hearing , 2005 .

[100]  Elizabeth M. Wenzel,et al.  Effect of Increasing System Latency on Localization of Virtual Sounds , 1999 .

[101]  Vikram Goyal Pro Java ME MMAPI: Mobile Media API for Java Micro Edition , 2006 .

[102]  Dan Gärdenfors,et al.  Audio games: new perspectives on game audio , 2004, ACE '04.

[103]  Paul A. Cairns,et al.  Measuring and defining the experience of immersion in games , 2008, Int. J. Hum. Comput. Stud..

[104]  Brenda Laurel,et al.  Computers as theatre , 1991 .

[105]  William Shottstaedt,et al.  Automatic counterpoint , 1989 .

[106]  Francis Rumsey,et al.  Computer Games and Multichannel Audio Quality Part 2 ' Evaluation of Time-Variant Audio Degradations Under Divided and Undivided Attention , 2003 .

[107]  H. Pashler The Psychology of Attention , 1997 .

[108]  Kristine Jørgensen,et al.  On the Functional Aspects of Computer Game Audio , 2006 .

[109]  Amyris Fernandez Fun Experience with Digital Games : a Model Proposition , 2007 .

[110]  D. Freeman,et al.  Soundscape : the school of sound lectures 1998-2001 , 2003 .

[111]  Lennart E. Nacke,et al.  Affective Ludology: Scientific Measurement of User Experience in Interactive Entertainment , 2009 .

[112]  M. Bull Sounding Out the City: Personal Stereos and the Management of Everyday Life , 2000 .

[113]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[114]  L. Salter,et al.  Spaces Speak, Are You Listening?: Experiencing Aural Architecture , 2006 .

[115]  Stephen A. Brewster,et al.  Correcting menu usability problems with sound , 1999, Behav. Inf. Technol..

[116]  N. Ravaja Contributions of Psychophysiology to Media Research: Review and Recommendations , 2004 .

[117]  S. Schachter The Interaction of Cognitive and Physiological Determinants of Emotional State , 1964 .

[118]  Ulrich Reiter,et al.  Perceived Quality in Game Audio , 2011 .

[119]  Lennart E. Nacke,et al.  Flow and immersion in first-person shooters: measuring the player's gameplay experience , 2008, Future Play.

[120]  R. Chapel Realtime Algorithmic Music Systems From Fractals and Chaotic Functions: Toward an Active Musical Instrument , 2003 .

[121]  Jürgen Meyer,et al.  ACOUSTICS AND THE PERFORMANCE OF MUSIC , 2024, Spring Conference 1981.

[122]  David Murphy Evaluation of spatial sound description in virtual environments , 1999 .

[123]  Curtis Roads,et al.  The Computer Music Tutorial , 1996 .

[124]  Heloir,et al.  The Uncanny Valley , 2019, The Animation Studies Reader.

[125]  Karolien Poels,et al.  Playability and player experience research , 2009 .

[126]  S. D. Jenkins,et al.  Comparing Thermographic, EEG, and Subjective Measures of Affective Experience During Simulated Product Interactions , 2009 .

[127]  James D. Ivory,et al.  The Effects of Technological Advancement and Violent Content in Video Games on Players’ Feelings of Presence, Involvement, Physiological Arousal, and Aggression , 2007 .

[128]  V. Ralph Algazi,et al.  The Use of Head-and-Torso Models for Improved Spatial Sound Synthesis , 2002 .

[129]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[130]  P. Toprac,et al.  Causing Fear, Suspense, and Anxiety Using Sound Design in Computer Games , 2011 .

[131]  Mark Grimshaw,et al.  Sound And Immersion In The First-Person Shooter , 2007, Int. J. Intell. Games Simul..

[132]  A. Combs,et al.  Absorption and Enjoyment of Music , 1988 .

[133]  Albert S. Bregman,et al.  The Auditory Scene. (Book Reviews: Auditory Scene Analysis. The Perceptual Organization of Sound.) , 1990 .

[134]  J. Sundberg,et al.  Overview of the KTH rule system for musical performance. , 2006 .

[135]  Andrew R. Brown,et al.  The Morph Table: A collaborative interface for musical interaction , 2007 .

[136]  Meyer Solomon,et al.  THE MECHANISM OF THE EMOTIONS1 , 1927 .

[137]  Herbert M. Schueller,et al.  Asthetik der Filmmusik , 1966 .

[138]  P. C. Pandey,et al.  The Journal of the Acoustical Society of America , 1939 .

[139]  Norbert Herber The Composition-Instrument: musical emergence and interaction , .

[140]  L. Clark,et al.  Gambling Near-Misses Enhance Motivation to Gamble and Recruit Win-Related Brain Circuitry , 2009, Neuron.

[141]  H. Schlosberg The description of facial expressions in terms of two dimensions. , 1952, Journal of experimental psychology.

[142]  Andrew P. Witkin,et al.  What Is Perceptual Organization For? , 1983, IJCAI.

[143]  Jaime Sánchez,et al.  Interactive 3D sound hyperstories for blind children , 1999, CHI '99.

[144]  Takashi Minato,et al.  Development of an Android Robot for Studying Human-Robot Interaction , 2004, IEA/AIE.

[145]  Jeff T. Larsen,et al.  The Agony of Victory and Thrill of Defeat , 2004, Psychological science.

[146]  P. Shand The Music of the Environment , 1974 .

[147]  Francis Rumsey,et al.  A scalable spatial sound rendering system. , 2001 .

[148]  K. Isbister,et al.  Chapter 14 – Physiological Measures for Game Evaluation* , 2008 .

[149]  W. James II.—WHAT IS AN EMOTION ? , 1884 .

[150]  Paul A. Cairns,et al.  From immersion to addiction in videogames , 2008, BCS HCI.

[151]  Laura A. Dabbish,et al.  Designing games with a purpose , 2008, CACM.

[152]  François Pachet,et al.  Musical Harmonization with Constraints: A Survey , 2004, Constraints.

[153]  W. Singer,et al.  Neuronal assemblies: necessity, signature and detectability , 1997, Trends in Cognitive Sciences.

[154]  Daniel Västfjäll,et al.  Better Presence and Performance in Virtual Environments by Improved Binaural Sound Rendering , 2002 .

[155]  J. Murray Hamlet on the Holodeck , 1997 .

[156]  Takayuki Kanda,et al.  Interactive Robots as Social Partners and Peer Tutors for Children: A Field Trial , 2004, Hum. Comput. Interact..

[157]  Donald A. Norman,et al.  Emotional design , 2004, UBIQ.

[158]  Knut Hartmann,et al.  Composition and Arrangement Techniques for Music in Interactive Immersive Environments , 2006 .

[159]  W. Kirchner Age differences in short-term retention of rapidly changing information. , 1958, Journal of experimental psychology.

[160]  Eddie Kohler,et al.  Real-time speech motion synthesis from recorded motions , 2004, SCA '04.

[161]  William V. Wright,et al.  A Theory of Fun for Game Design , 2004 .

[162]  Karen Collins,et al.  Sound in Electronic Gambling Machines: A Review of the Literature and its Relevance to Game Sound , 2011 .

[163]  G. Calleja Digital games as designed experience : reframing the concept of immersion , 2007 .

[164]  C. Anderson,et al.  Violent Video Games and Hostile Expectations: A Test of the General Aggression Model , 2002 .

[165]  Satu Jumisko-Pyykkö,et al.  Watch, Press, and Catch - Impact of Divided Attention on Requirements of Audiovisual Quality , 2007, HCI.

[166]  David A. Burgess Techniques for low cost spatial audio , 1992, UIST '92.

[167]  S. Wolfson,et al.  The effects of sound and colour on responses to a computer game , 2000, Interact. Comput..

[168]  S. Shimojo,et al.  Illusions: What you see is what you hear , 2000, Nature.

[169]  S. Freud The Standard Edition of the Complete Psychological Works of Sigmund Freud , 1953 .

[170]  Andy Farnell Behaviour, Structure and Causality in Procedural Audio1 , 2011 .

[171]  R. Shiffrin,et al.  Can attention be allocated to sensory modalities? , 1974 .

[172]  P. McFedries,et al.  Filling the gaps , 2002, IEEE Spectrum.

[173]  Raine Kajastila,et al.  Localization Cues Affect Emotional Judgments – Results from a User Study on Scary Sound , 2009 .

[174]  Frans Mäyrä,et al.  Fundamental Components of the Gameplay Experience: Analysing Immersion , 2005, DiGRA Conference.

[175]  Hildegard Westerkamp,et al.  Listening and soundmaking : a study of music-as-environment , 1988 .

[176]  M. R. Schroeder,et al.  Digital simulation of sound transmission in reverberant spaces (part 1) , 1969 .

[177]  Katherine Isbister,et al.  Game Usability - Advice from the Experts for Advancing the Player Experience , 2008 .

[178]  Kai Kuikkaniemi,et al.  Position Paper for BRAINPLAY ’ 07 : Playing with Your Brain Workshop : Progressive System Architecture for Building Emotionally Adaptive Games , 2007 .

[179]  Nilli Lavie Capacity limits in selective attention: Behavioral evidence and implications for neural activity , 2001 .

[180]  General , 1970 .

[181]  Pavel Zahorik,et al.  Presence as Being-in-the-World , 1998, Presence.

[182]  Guillaume Roux-Girard Listening to Fear: A Study of Sound in Horror Computer Games , 2011 .

[183]  Karl F. MacDorman,et al.  Sensitivity to the proportions of faces that vary in human likeness , 2008, Comput. Hum. Behav..

[184]  B. R. Ghioto Action and reaction. , 1971, Occupational health nursing.

[185]  Licinio Roque,et al.  A Sociotechnical Conjecture about the Context and Development of Multiplayer Online Game Experiences , 2005, DiGRA Conference.

[186]  D Wilkiemeyer,et al.  The man who Knew too Much , 1978, Nursing.

[187]  John P. Charlton,et al.  Differentiating computer-related addictions and high engagement , 2004 .

[188]  Martin H. Levinson,et al.  Happiness: The Science Behind Your Smile , 2005 .

[189]  Takashi Minato,et al.  Generating natural motion in an android by mapping human motion , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[190]  J. Lagopoulos Electrodermal activity , 2007, Acta Neuropsychiatrica.

[191]  M. Pépin,et al.  Effect of Playing a Video Game on a Measure of Spatial Visualization , 1986, Perceptual and motor skills.

[192]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[193]  Søren Bech,et al.  Interaction Between Audio-Visual Factors in a Home Theater System: Definition of Subjective Attributes , 1995 .

[194]  T. Risley,et al.  The effects of play materials on social play. , 1973, Journal of applied behavior analysis.

[195]  Joseph E LeDoux The emotional brain , 1996 .

[196]  Karl F. MacDorman,et al.  Human emotion and the uncanny valley: A GLM, MDS, and Isomap analysis of robot video ratings , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[197]  Jean-Pierre Lecanuet Prenatal auditory experience , 1996 .

[198]  Jeff T. Larsen,et al.  Can people feel happy and sad at the same time? , 2001, Journal of personality and social psychology.

[199]  Barry Arons,et al.  A Review of The Cocktail Party Effect , 1992 .

[200]  Susan Persky,et al.  The Independent and Interactive Effects of Embodied-Agent Appearance and Behavior on Self-Report, Cognitive, and Behavioral Markers of Copresence in Immersive Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[201]  Kemal Ebcioglu,et al.  An Expert System for Harmonizing Chorales in the Style of J. S. Bach , 1990, J. Log. Program..

[202]  Davide Rocchesso,et al.  Sound to Sense - Sense to Sound: A state of the art in Sound and Music Computing , 2008 .

[203]  K. Mathiak,et al.  Toward brain correlates of natural behavior: fMRI during violent video games , 2006, Human brain mapping.

[204]  J. Russell A circumplex model of affect. , 1980 .

[205]  Johnny Wingstedt,et al.  Making music mean : on functions of, and knowledge about, narrative music in multimedia , 2008 .

[206]  K. MacDorman,et al.  Subjective Ratings of Robot Video Clips for Human Likeness, Familiarity, and Eeriness: An Exploration of the Uncanny Valley , 2006 .

[207]  D. Norman Emotional design : why we love (or hate) everyday things , 2004 .

[208]  Alan J. Dix,et al.  Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me (ACE) , 2005, DiGRA Conference.

[209]  Gareth Schott,et al.  A Conceptual Framework for the Analysis of First-Person Shooter Audio and its Potential Use for Game Engines , 2008, Int. J. Comput. Games Technol..

[210]  Andrew Hargadon,et al.  The pleasure principle: immersion, engagement, flow , 2000, HYPERTEXT '00.

[211]  D. Kahneman,et al.  Attention and Effort , 1973 .

[212]  Heinrich K. Taube,et al.  Notes from the Metalevel: An Introduction to Algorithmic Music Composition , 2006 .

[213]  Nigel Papworth,et al.  Beowulf: an audio mostly game , 2007, ACE '07.

[214]  Max V. Mathews,et al.  Current directions in computer music research , 1989 .

[215]  S. Paradiso The Emotional Brain: The Mysterious Underpinnings of Emotional Life , 1998 .

[216]  Jesper Juul Half-Real: Video Games between Real Rules and Fictional Worlds , 2005 .

[217]  Francis Rumsey,et al.  Computer Games and Multichannel Audio Quality - The Effect of Division of Attention between Auditory and Visual Modalities , 2003 .

[218]  S. Jumisko-Pyykko,et al.  Produced Quality is Not Perceived Quality - A Qualitative Approach to Overall Audiovisual Quality , 2007, 2007 3DTV Conference.

[219]  A. Galloway,et al.  Gaming: Essays On Algorithmic Culture , 2006 .

[220]  Frans Mäyrä,et al.  Designing sound for a pervasive mobile game , 2005, ACE '05.

[221]  M. Zyda,et al.  Introducing Emotion into Military Simulation and Video Game Design America's Army: Operations and VIRTE , 2002, GAME-ON.

[222]  J. Gackenbach The Relationship Between Perceptions of Video Game Flow and Structure , 2008 .

[223]  P. Lang The varieties of emotional experience: a meditation on James-Lange theory. , 1994, Psychological review.

[224]  Christopher Frauenberger,et al.  Ears ))): a methodological framework for auditory display design , 2007, CHI Extended Abstracts.

[225]  Geraint A. Wiggins,et al.  AI Methods for Algorithmic Composition: A Survey, a Critical View and Future Prospects , 1999 .

[226]  M. Bradley,et al.  Emotion and motivation. , 2007 .

[227]  Lennart E. Nacke,et al.  Affective Ludology, Flow and Immersion in a First- Person Shooter: Measurement of Player Experience , 2009, ArXiv.

[228]  Olli Sotamaa,et al.  The Player's Game: Towards Understanding Player Production Among Computer Game Cultures , 2009 .

[229]  D. H. Warren,et al.  The role of visual-auditory “compellingness” in the ventriloquism effect: Implications for transitivity among the spatial senses , 1981, Perception & psychophysics.

[230]  Richard D. Lane,et al.  The study of emotion from the perspective of cognitive neuroscience , 2000 .

[231]  David Murphy,et al.  Spatial Sound Enhancing Virtual Story Telling , 2001, International Conference on Virtual Storytelling.

[232]  Hugo Fastl,et al.  Psychoacoustics: Facts and Models , 1990 .

[233]  D. Dooley Social Research Methods , 1990 .

[234]  C. Lange THE MECHANISM OF THE EMOTIONS , 2005 .

[235]  Anker Helms Jørgensen Marrying HCI/Usability and computer games: a preliminary look , 2004, NordiCHI '04.

[236]  R. M. Schafer,et al.  The Soundscape: Our Sonic Environment and the Tuning of the World , 1993 .

[237]  Karin Bijsterveld,et al.  Mechanical Sound: Technology, Culture, and Public Problems of Noise in theTwentieth Century , 2008 .

[238]  Lennart E. Nacke,et al.  More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game , 2010, Interact. Comput..

[239]  M. Csíkszentmihályi Beyond boredom and anxiety , 1975 .

[240]  Gordon Calleja,et al.  Revising Immersion: A Conceptual Model for the Analysis of Digital Game Involvement , 2007, DiGRA Conference.

[241]  Rolf Nordahl Self-induced Footsteps Sounds in Virtual Reality: Latency, Recognition, Quality and Presence , 2005 .

[242]  Thomas B. Sheridan Further Musings on the Psychophysics of Presence , 1996, Presence: Teleoperators & Virtual Environments.

[243]  Andrew Polaine,et al.  The flow principle in interactivity , 2005 .

[244]  Ian P. Howard,et al.  Human visual orientation , 1982 .

[245]  Karen Collins,et al.  Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design , 2008 .

[246]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[247]  Angela Tinwell Uncanny as Usability Obstacle , 2009, HCI.

[248]  M. Whitton,et al.  Effect of latency on presence in stressful virtual environments , 2003, IEEE Virtual Reality, 2003. Proceedings..

[249]  Ulrich Reiter,et al.  Influence of Interaction on Perceived Quality in Audiovisual Applications: Evaluation of Cross-Modal Influence , 2007 .

[250]  Daniel Hug,et al.  New Wine in New Skins: Sketching the Future of Game Sound Design , 2011 .

[251]  Gareth Schott,et al.  Situating Gaming as a Sonic Experience: The acoustic ecology of First-Person Shooters , 2007, DiGRA Conference.

[252]  M. Paradiso,et al.  Neuroscience: Exploring the brain, 3rd ed. , 2007 .

[253]  George A. Gescheider,et al.  Simultaneous auditory and tactile information processing , 1975 .

[254]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[255]  C. Anderson,et al.  The effect of video game violence on physiological desensitization to real-life violence. , 2007 .

[256]  P. Ekman An argument for basic emotions , 1992 .

[257]  A Freese,et al.  The power of sound. , 1970, Tic.

[258]  S. Shimojo,et al.  Visual illusion induced by sound. , 2002, Brain research. Cognitive brain research.

[259]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[260]  Jim R. Parker,et al.  Audio Interaction in Computer Mediated Games , 2008, Int. J. Comput. Games Technol..

[261]  J. Panksepp Affective Neuroscience: The Foundations of Human and Animal Emotions , 1998 .

[262]  Andrew K Przybylski,et al.  The Motivating Role of Violence in Video Games , 2009, Personality & social psychology bulletin.

[263]  Kristine Jørgensen,et al.  Audio and Gameplay: An Analysis of PvP Battlegrounds in World of Warcraft , 2008, Game Stud..

[264]  Jeff T. Larsen,et al.  Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii. , 2003, Psychophysiology.

[265]  Andrew R. Brown,et al.  Investigating Morphing Algorithms for Generative Music , 2005 .

[266]  Donald A. Norman,et al.  Emotion & design: attractive things work better , 2002, INTR.

[267]  R. Blake,et al.  Neural strength of visual attention gauged by motion adaptation , 1999, Nature Neuroscience.

[268]  Takayuki Kanda,et al.  My robotic doppelgänger - a critical look at the Uncanny Valley , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[269]  Karin Bijsterveld,et al.  The diabolical symphony of the mechanical age. Technology and symbolism of sound in European and North American noise abatement campaign, 1900-40 , 2003 .

[270]  Andy Hunt,et al.  Guest Editors' Introduction: An Introduction to Interactive Sonification , 2005, IEEE Multim..

[271]  M. F. Washburn,et al.  Grundriss der Psychologie , 1894, Nature.

[272]  John Dack,et al.  Soundscape: the school of sound lectures. , 2005 .

[273]  A. Bryman Social Research Methods , 2001 .

[274]  Aaron Marks,et al.  Game Development Essentials: Game Audio Development , 2008 .

[275]  Alexander Refsum Jensenius,et al.  The Drum Pants , 2006 .

[276]  P. Venables,et al.  Direct measurement of skin conductance: a proposal for standardization. , 1971, Psychophysiology.

[277]  Makoto Otani,et al.  A fast calculation method of the head-related transfer functions for multiple source points based on the boundary element method , 2003 .

[278]  Robert C. Seamans,et al.  Action and reaction , 1969 .

[279]  K. Isbister,et al.  Beyond game design : Nine steps towards better videogames , 2009 .

[280]  W. Cannon The James-Lange theory of emotions: a critical examination and an alternative theory. By Walter B. Cannon, 1927. , 1927, The American journal of psychology.

[281]  Steven R. Livingstone Changing Musical Emotion through Score and Performance with a Computational Rule System , 2008 .

[282]  Kalle Jegers,et al.  Elaborating eight elements of fun: Supporting design of pervasive player enjoyment , 2009, CIE.

[283]  M. Bradley,et al.  Emotion and motivation I: defensive and appetitive reactions in picture processing. , 2001, Emotion.

[284]  Elena Karahanna,et al.  Time Flies When You're Having Fun: Cognitive Absorption and Beliefs About Information Technology Usage , 2000, MIS Q..

[285]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[286]  Jean-Marc Jot,et al.  Digital Delay Networks for Designing Artificial Reverberators , 1991 .

[287]  Gray Hodgkinson The seduction of realism , 2009, SIGGRAPH ASIA '09.

[288]  Richard L. Hazlett,et al.  Measuring emotional valence during interactive experiences: boys at video game play , 2006, CHI.

[289]  Peta Wyeth,et al.  GameFlow: a model for evaluating player enjoyment in games , 2005, CIE.

[290]  Axel Berndt,et al.  Expressive Musical Timing , 2009 .

[291]  Mathis Löthe Ein wissensbasiertes Verfahren zur Komposition von frühklassischen Menuetten , 2003 .

[292]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[293]  W. James The Physical Basis of Emotion. , 1994 .

[294]  Mark Grimshaw,et al.  Uncanny behaviour in survival horror games , 2010 .

[295]  John A. Sloboda,et al.  Musical beginnings : origins and development of musical competence , 1996 .

[296]  Paul Dourish,et al.  Affect: from information to interaction , 2005, Critical Computing.

[297]  Antti Pirhonen,et al.  Same sound – Different meanings : A Novel Scheme for Modes of Listening , 2010 .

[298]  Jesse Schell,et al.  The Art of Game Design: A book of lenses , 2019 .

[299]  Heinrich K. Taube Notes from the Metalevel: An Introduction to Computer Composition , 2004 .

[300]  Wolfram Menzel,et al.  Learning musical structure and style with neural networks , 1998 .

[301]  Pippin Barr,et al.  Video Game Values: Play as Human-Computer Interaction , 2008 .

[302]  Phoebe Sengers,et al.  The disenchantment of affect , 2008, Personal and Ubiquitous Computing.

[303]  Phoebe Sengers,et al.  Staying open to interpretation: engaging multiple meanings in design and evaluation , 2006, DIS '06.

[304]  D. Watson,et al.  On the Dimensional and Hierarchical Structure of Affect , 1999 .

[305]  Mark Grimshaw,et al.  Bridging the uncanny: an impossible traverse? , 2009, MindTrek '09.

[306]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[307]  L. Nadel,et al.  Cognitive Neuroscience of Emotion. Series in Affective Science , 2000 .

[308]  Lynn Westbrook Human perspectives in the Internet society: Culture, psychology and gender , 2007, J. Assoc. Inf. Sci. Technol..

[309]  William Huber,et al.  Ghastly multiplication: Fatal Frame II and the Videogame Uncanny , 2007, DiGRA Conference.

[310]  Andrew K. Przybylski,et al.  The Motivational Pull of Video Games: A Self-Determination Theory Approach , 2006 .

[311]  N. Lazzaro Why We Play : Affect and the Fun of Games—Designing Emotions for Games, Entertainment Interfaces, and Interactive Products , 2012 .

[312]  Chris Bateman,et al.  21st Century Game Design , 2005 .

[313]  A. Damasio The Feeling of What Happens: Body and Emotion in the Making of Consciousness , 1999 .

[314]  Edward A. Vessel,et al.  Perceptual Pleasure and the Brain , 2006 .

[315]  J Edworthy,et al.  Improving Auditory Warning Design: Relationship between Warning Sound Parameters and Perceived Urgency , 1991, Human factors.

[316]  Sascha Mahlke,et al.  Studying antecedents of emotional experiences in interactive contexts , 2007, CHI.

[317]  Karin Bijsterveld,et al.  The Diabolical Symphony of the Mechanical Age: , 2001 .

[318]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[319]  Holger Theisel,et al.  Adaptive Musical Expression from Automatic Realtime Orchestration and Performance , 2008, ICIDS.

[320]  A. Ohman,et al.  Emotion drives attention: detecting the snake in the grass. , 2001, Journal of experimental psychology. General.

[321]  D. Västfjäll,et al.  Emotional responses to music: the need to consider underlying mechanisms. , 2008, The Behavioral and brain sciences.

[322]  Guerino Mazzola,et al.  The Topos of Music: Geometric Logic of Concepts, Theory, and Performance , 2002 .

[323]  Karl F. MacDorman,et al.  Too real for comfort? Uncanny responses to computer generated faces , 2009, Comput. Hum. Behav..

[324]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..

[325]  Mark Grimshaw The resonating spaces of First-Person Shooter games , 2007 .

[326]  David J. Miller,et al.  Using a games console in the primary classroom: Effects of 'Brain Training' programme on computation and self-esteem , 2010, Br. J. Educ. Technol..

[327]  W. James,et al.  What Is an Emotion , 1977 .

[328]  Laurent Romary,et al.  Visual Salience and Perceptual Grouping in Multimodal Interactivity , 2001 .

[329]  Jonathan Steuer,et al.  Defining virtual reality: dimensions determining telepresence , 1992 .

[330]  F. Holmes Atwater,et al.  Inducing Altered States of Consciousness with Binaural Beat Technology , 1997 .

[331]  Hitendra K. Pillay,et al.  An Investigation of Cognitive Processes Engaged in by Recreational Computer Game Players , 2002 .

[332]  J. Vaidya,et al.  The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence , 1999 .

[333]  C. Ferguson Evidence for publication bias in video game violence effects literature: A meta-analytic review , 2007 .

[334]  Alan J. Dix,et al.  Using frustration in the design of adaptive videogames , 2004, ACE '04.

[335]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[336]  D H Ashmead,et al.  Auditory perception of walls via spectral variations in the ambient sound field. , 1999, Journal of rehabilitation research and development.

[337]  Paul A. Cairns,et al.  A grounded investigation of game immersion , 2004, CHI EA '04.

[338]  Shrikanth Narayanan,et al.  Interplay between linguistic and affective goals in facial expression during emotional utterances , 2006 .

[339]  Judy Edworthy Does sound help us to work better with machines? A commentary on Rauterberg's paper ‘About the importance of auditory alarms during the operation of a plant simulator’ , 1998 .

[340]  Manfred R. Schroeder,et al.  Natural Sounding Artificial Reverberation , 1962 .

[341]  Lennart E. Nacke,et al.  Log Who's Playing: Psychophysiological Game Analysis Made Easy through Event Logging , 2008, Fun and Games.

[342]  Barry Truax Sound in Context: Soundscape Research and Composition at Simon Fraser University , 1995, ICMC.

[343]  Darryl Charles,et al.  Toward an understanding of flow in video games , 2008, CIE.

[344]  Yifan Wang,et al.  Exploring the Uncanny Valley with Japanese Video Game Characters , 2007, DiGRA Conference.

[345]  Ville Pulkki,et al.  Spatial sound generation and perception by amplitude panning techniques , 2001 .

[346]  Mel Slater,et al.  Building Characters: Lessons Drawn from Virtual Environments , 2005 .

[347]  Joel Chadabe Interactive music composition and performance system , 1985 .

[348]  Leon Cohen,et al.  The history of noise [on the 100th anniversary of its birth] , 2005, IEEE Signal Processing Magazine.

[349]  On the quality of experience : A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments , 2003 .

[350]  Eliza Ward,et al.  An Introduction to Acoustic Ecology , 2001 .

[351]  D. Kort,et al.  D3.3 : Game Experience Questionnaire:development of a self-report measure to assess the psychological impact of digital games , 2007 .

[352]  L Back,et al.  The auditory culture reader , 2003 .

[353]  Mags Adams,et al.  Sustainable Soundscapes: Noise Policy and the Urban Experience , 2006 .

[354]  J. Graftieaux [The uncanny]. , 2011, Annales francaises d'anesthesie et de reanimation.

[355]  Samuel Adler,et al.  The Study of Orchestration , 1982 .

[356]  Erik Champion,et al.  Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game Using Biofeedback , 2007, DiGRA Conference.

[357]  R. M. Schafer,et al.  The tuning of the world , 1977 .

[358]  R. Rosner Computer software , 1978, Nature.