A closed-loop, music-based brain-computer interface for emotion mediation

Emotions play a critical role in rational and intelligent behavior; a better fundamental knowledge of them is indispensable for understanding higher order brain function. We propose a non-invasive brain-computer interface (BCI) system to feedback a person’s affective state such that a closed-loop interaction between the participant’s brain responses and the musical stimuli is established. We realized this concept technically in a functional prototype of an algorithm that generates continuous and controllable patterns of synthesized affective music in real-time, which is embedded within a BCI architecture. We evaluated our concept in two separate studies. In the first study, we tested the efficacy of our music algorithm by measuring subjective affective responses from 11 participants. In a second pilot study, the algorithm was embedded in a real-time BCI architecture to investigate affective closed-loop interactions in 5 participants. Preliminary results suggested that participants were able to intentionally modulate the musical feedback by self-inducing emotions (e.g., by recalling memories), suggesting that the system was able not only to capture the listener’s current affective state in real-time, but also potentially provide a tool for listeners to mediate their own emotions by interacting with music. The proposed concept offers a tool to study emotions in the loop, promising to cast a complementary light on emotion-related brain research, particularly in terms of clarifying the interactive, spatio-temporal dynamics underlying affective processing in the brain.

[1]  Vincent Persichetti Twentieth-Century Harmony: Creative Aspects and Practice , 1961 .

[2]  N. Geschwind Language and the brain. , 1972, Scientific American.

[3]  R. Graham Biofeedback and the Arts: Results of Early Experiments , 1976 .

[4]  J. Russell A circumplex model of affect. , 1980 .

[5]  Ernst Fernando Lopes Da Silva Niedermeyer,et al.  Electroencephalography, basic principles, clinical applications, and related fields , 1982 .

[6]  J. Rothgeb,et al.  Introduction to the Theory of Heinrich Schenker: The Nature of the Musical Work of Art , 1982 .

[7]  W. Ray,et al.  EEG correlates of emotional tasks related to attentional demands. , 1985, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[8]  R. Homan,et al.  Cerebral location of international 10-20 system electrode placement. , 1987, Electroencephalography and clinical neurophysiology.

[9]  M. Schmuckler Expectation in music: Investigation of melodic and harmonic processes. , 1989 .

[10]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[11]  Bruno Nettl,et al.  Music, Culture, and Experience: Selected Papers of John Blacking , 1995 .

[12]  C. Krumhansl An exploratory study of musical emotions and psychophysiology. , 1997, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[13]  Matthias M. Müller,et al.  Processing of affective pictures modulates right-hemispheric gamma band EEG activity , 1999, Clinical Neurophysiology.

[14]  S. Ashida The effect of reminiscence music therapy sessions on changes in depressive symptoms in elderly persons with dementia. , 2000, Journal of music therapy.

[15]  Joseph E LeDoux Emotion Circuits in the Brain , 2000 .

[16]  J. Sloboda,et al.  Music and emotion: Theory and research , 2001 .

[17]  Michel Tuan Pham,et al.  When Arousal Influences Ad Evaluation and Valence Does Not (and Vice Versa) , 2001 .

[18]  J. Sloboda,et al.  Emotions in everyday listening to music. , 2001 .

[19]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[20]  Matthias M. Müller,et al.  Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials , 2001, Clinical Neurophysiology.

[21]  I. Cross Music, Cognition, Culture, and Evolution , 2001, Annals of the New York Academy of Sciences.

[22]  R. Dolan,et al.  Emotion, Cognition, and Behavior , 2002, Science.

[23]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[24]  Rosalind W. Picard Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[25]  J. Gross,et al.  Thinking makes it so: A social cognitive neuroscience approach to emotion regulation. , 2004 .

[26]  Eduardo Reck Miranda,et al.  Plymouth brain-Computer Music Interface Project: Intelligent assistive Technology for Music-Making , 2005, ICMC.

[27]  Thilo Hinterberger,et al.  Parametric orchestral sonification of EEG in real time , 2005, IEEE MultiMedia.

[28]  Ya. I. Levin “Brain music” in the treatment of patients with insomnia , 1998, Neuroscience and Behavioral Physiology.

[29]  M. Balconi,et al.  EEG correlates (event-related desynchronization) of emotional face elaboration: A temporal analysis , 2006, Neuroscience Letters.

[30]  T. Baumgartner,et al.  From emotion perception to emotion experience: emotions evoked by pictures and classical music. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[31]  S. Trehub,et al.  Infant music perception: Domain-general or domain-specific mechanisms? , 2006, Cognition.

[32]  Aniruddh D. Patel Music, Language, and the Brain , 2007 .

[33]  P. Gomez,et al.  Relationships between musical structure and psychophysiological measures of emotion. , 2007, Emotion.

[34]  W. Magee Music as a diagnostic tool in low awareness states: Considering limbic responses , 2007, Brain injury.

[35]  A. Maratos,et al.  Music therapy for depression. , 2008, The Cochrane database of systematic reviews.

[36]  J. Gross,et al.  Cognitive Emotion Regulation , 2008, Current directions in psychological science.

[37]  J. Wolpaw,et al.  Brain–computer interfaces in neurological rehabilitation , 2008, The Lancet Neurology.

[38]  M. Balconi,et al.  Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[39]  E. Campana,et al.  COMPUTER-GENERATING EMOTIONAL MUSIC : THE DESIGN OF AN AFFECTIVE MUSIC ALGORITHM , 2008 .

[40]  P. Juslin Emotional responses to music: The need to consider underlying mechanisms , 2008, Behavioral and Brain Sciences.

[41]  Yi-Hsuan Yang,et al.  A Regression Approach to Music Emotion Recognition , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[42]  M. Arns,et al.  Efficacy of Neurofeedback Treatment in ADHD: The Effects on Inattention, Impulsivity and Hyperactivity: A Meta-Analysis , 2009, Clinical EEG and neuroscience.

[43]  Anton Nijholt,et al.  Emotional brain-computer interfaces , 2009, ACII.

[44]  S. Koole The psychology of emotion regulation: An integrative review , 2009 .

[45]  J. Touchon,et al.  Effect of Music Therapy on Anxiety and Depression in Patients with Alzheimer’s Type Dementia: Randomised, Controlled Study , 2009, Dementia and Geriatric Cognitive Disorders.

[46]  M. Balconi,et al.  Brain oscillations and BIS/BAS (behavioral inhibition/activation system) effects on processing masked emotional cues. ERS/ERD and coherence measures of alpha band. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[47]  Thanh Nhan Nguyen,et al.  Music Therapy to Reduce Pain and Anxiety in Children With Cancer Undergoing Lumbar Puncture: A Randomized Clinical Trial , 2010, Journal of pediatric oncology nursing : official journal of the Association of Pediatric Oncology Nurses.

[48]  Anil K. Seth,et al.  A MATLAB toolbox for Granger causal connectivity analysis , 2010, Journal of Neuroscience Methods.

[49]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[50]  Niels Birbaumer,et al.  Real-time support vector classification and feedback of multiple emotional brain states , 2011, NeuroImage.

[51]  Affecticon: emotion-based icons for music retrieval. , 2011, IEEE computer graphics and applications.

[52]  Scott Makeig,et al.  First Demonstration of a Musical Emotion BCI , 2011, ACII.

[53]  Michael,et al.  Affective Computing and Intelligent Interaction , 2011, Lecture Notes in Computer Science.

[54]  Olga Sourina,et al.  Real-time EEG-based emotion recognition for music therapy , 2011, Journal on Multimodal User Interfaces.

[55]  K. Adalarasu,et al.  A REVIEW ON INFLUENCE OF MUSIC ON BRAIN ACTIVITY USING SIGNAL PROCESSING AND IMAGING SYSTEM , 2011 .

[56]  György Fazekas,et al.  Music Emotion Recognition: From Content- to Context-Based Models , 2012, CMMR.

[57]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[58]  Richard Kronland-Martinet,et al.  From Sounds to Music and Emotions , 2012, Lecture Notes in Computer Science.

[59]  Miyoung Kim,et al.  A Review on the Computational Methods for Emotional State Estimation from the Human EEG , 2013, Comput. Math. Methods Medicine.

[60]  Hiroshi Nishiura,et al.  Age-Dependent Estimates of the Epidemiological Impact of Pandemic Influenza (H1N1-2009) in Japan , 2013, Comput. Math. Methods Medicine.

[61]  G. McArthur,et al.  Validation of the Emotiv EPOC® EEG gaming system for measuring research quality auditory ERPs , 2013, PeerJ.

[62]  C. Moritz,et al.  Affective Brain-Computer Interfaces As Enabling Technology for Responsive Psychiatric Stimulation. , 2014, Brain computer interfaces.

[63]  Rajesh P. N. Rao,et al.  Brain–computer interfaces: a powerful tool for scientific inquiry , 2014, Current Opinion in Neurobiology.

[64]  A. Nijholt,et al.  A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges , 2014 .

[65]  Gordon Cheng,et al.  Augmenting Affect from Speech with Generative Music , 2015, CHI Extended Abstracts.

[66]  Sergio I. Giraldo,et al.  Musical neurofeedback for treating depression in elderly people , 2015, Front. Neurosci..

[67]  G. McArthur,et al.  Validation of the Emotiv EPOC EEG system for research quality auditory event-related potentials in children , 2015, PeerJ.

[68]  Frederic M. Stoll,et al.  The Effects of Cognitive Control and Time on Frontal Beta Oscillations. , 2016, Cerebral cortex.

[69]  J. Ket,et al.  Music therapy for depression. , 2017, The Cochrane database of systematic reviews.

[70]  F. Darvas,et al.  The Encephalophone: A Novel Musical Biofeedback Device using Conscious Control of Electroencephalogram (EEG) , 2017, Front. Hum. Neurosci..

[71]  Gordon Cheng,et al.  A closed-loop brain-computer music interface for continuous affective interaction , 2017, 2017 International Conference on Orange Technologies (ICOT).