Music mood and human emotion recognition based on physiological signals: a systematic review

Scientists and researchers have tried to establish a bond between the emotions conveyed and the subsequent mood perceived in a person. Emotions play a major role in terms of our choices, preferences, and decision-making. Emotions appear whenever a person perceives a change in their surroundings or within their body. Since early times, a considerable amount of effort has been made in the field of emotion detection and mood estimation. Listening to music forms a major part of our daily life. The music we listen to, the emotions it induces, and the resulting mood are all interrelated in ways we are unbeknownst to, and our survey is entirely based on these two areas of research. Differing viewpoints on this issue have led to the proposal of different ways of emotion annotation, model training, and result visualization. This paper provides a detailed review of the methods proposed in music mood recognition. It also discusses the different sensors that have been utilized to acquire various physiological signals. This paper will focus upon the datasets created and reused, different classifiers employed to obtain results with higher accuracy, features extracted from the acquired signals, and music along with an attempt to determine the exact features and parameters that will help in improving the classification process. It will also investigate several techniques to detect emotions and the different music models used to assess the music mood. This review intends to answer the questions and research issues in identifying human emotions and music mood to provide a greater insight into this field of interest and develop a better understanding to comprehend and answer the perplexing problems that surround us.

[1]  I. Peretz,et al.  Singing in the Brain: Independence of Lyrics and Tunes , 1998 .

[2]  Olga Sourina,et al.  Real-time EEG-based emotion monitoring using stable features , 2015, The Visual Computer.

[3]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[4]  U. Rajendra Acharya,et al.  A deep convolutional neural network model for automated identification of abnormal EEG signals , 2018, Neural Computing and Applications.

[5]  Shilpa Bagdare,et al.  Music and consumption experience: a review , 2011 .

[6]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[7]  Mohsen Naji,et al.  Classification of Music-Induced Emotions Based on Information Fusion of Forehead Biosignals and Electrocardiogram , 2014, Cognitive Computation.

[8]  U. Rajendra Acharya,et al.  Characterization of focal EEG signals: A review , 2019, Future Gener. Comput. Syst..

[9]  Jonathan Foote,et al.  Content-based retrieval of music and audio , 1997, Other Conferences.

[10]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.

[11]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[12]  Yi-Hsuan Yang,et al.  Machine Recognition of Music Emotion: A Review , 2012, TIST.

[13]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[14]  Dip Paul,et al.  A Survey of Music Recommendation Systems with a Proposed Music Recommendation System , 2019, Advances in Intelligent Systems and Computing.

[15]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[16]  Antonio Fernández-Caballero,et al.  Influence of Tempo and Rhythmic Unit in Musical Emotion Regulation , 2016, Front. Comput. Neurosci..

[17]  Mohsen Naji,et al.  Emotion classification during music listening from forehead biosignals , 2015, Signal Image Video Process..

[18]  Franco Turini,et al.  A Survey of Methods for Explaining Black Box Models , 2018, ACM Comput. Surv..

[19]  Ioannis Patras,et al.  AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups , 2017, IEEE Transactions on Affective Computing.

[20]  Mohammad-Parsa Hosseini,et al.  A Review on Machine Learning for EEG Signal Processing in Bioengineering , 2020, IEEE Reviews in Biomedical Engineering.

[21]  R. Mccraty,et al.  The effects of different types of music on mood, tension, and mental clarity. , 1998, Alternative therapies in health and medicine.

[22]  İbrahim Türkoğlu,et al.  Feature Selection with Sequential Forward Selection Algorithm from Emotion Estimation based on EEG Signals , 2019 .

[23]  Shenglan Liu,et al.  Deep attention based music genre classification , 2020, Neurocomputing.

[24]  Leontios J. Hadjileontiadis,et al.  EEG-Based Classification of Music Appraisal Responses Using Time-Frequency Analysis and Familiarity Ratings , 2013, IEEE Transactions on Affective Computing.

[25]  Katharina Morik,et al.  A Benchmark Dataset for Audio Classification and Clustering , 2005, ISMIR.

[26]  J. Russell Culture and the categorization of emotions. , 1991, Psychological bulletin.

[27]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[28]  T. Eerola,et al.  A comparison of the discrete and dimensional models of emotion in music , 2011 .

[29]  Yong-Jin Liu,et al.  Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals , 2018, IEEE Transactions on Affective Computing.

[30]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[31]  George Tzanetakis,et al.  Musical genre classification of audio signals , 2002, IEEE Trans. Speech Audio Process..

[32]  Pasin Israsena,et al.  Real-Time EEG-Based Happiness Detection System , 2013, TheScientificWorldJournal.

[33]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[34]  Ralph G Andrzejak,et al.  Nonrandomness, nonlinear dependence, and nonstationarity of electroencephalographic recordings from epilepsy patients. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[35]  John Atkinson,et al.  Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers , 2016, Expert Syst. Appl..

[36]  P. Ekman An argument for basic emotions , 1992 .

[37]  N. Scaringella,et al.  Automatic genre classification of music content: a survey , 2006, IEEE Signal Process. Mag..

[38]  Daniel P. W. Ellis,et al.  Please Scroll down for Article Journal of New Music Research a Web-based Game for Collecting Music Metadata a Web-based Game for Collecting Music Metadata , 2022 .

[39]  P. Laukka,et al.  Expression, Perception, and Induction of Musical Emotions: A Review and a Questionnaire Study of Everyday Listening , 2004 .

[40]  Hui Zhang,et al.  The PMEmo Dataset for Music Emotion Recognition , 2018, ICMR.

[41]  Lie Lu,et al.  Music type classification by spectral contrast feature , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[42]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[43]  Lie Lu,et al.  Automatic mood detection and tracking of music audio signals , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[44]  Anthony G. Hudetz,et al.  Functional and Topological Conditions for Explosive Synchronization Develop in Human Brain Networks with the Onset of Anesthetic-Induced Unconsciousness , 2016, Front. Comput. Neurosci..

[45]  G. Saha,et al.  Recognition of emotions induced by music videos using DT-CWPT , 2013, 2013 Indian Conference on Medical Informatics and Telemedicine (ICMIT).

[46]  N. Birbaumer,et al.  BCI2000: a general-purpose brain-computer interface (BCI) system , 2004, IEEE Transactions on Biomedical Engineering.

[47]  O. E. Dror,et al.  The Cannon–Bard Thalamic Theory of Emotions: A Brief Genealogy and Reappraisal , 2014 .

[48]  J. Russell,et al.  An approach to environmental psychology , 1974 .

[49]  G.B. Moody,et al.  The impact of the MIT-BIH Arrhythmia Database , 2001, IEEE Engineering in Medicine and Biology Magazine.

[50]  Michael H. Thaut,et al.  Rhythm, Music, and the Brain: Scientific Foundations and Clinical Applications , 2005 .

[51]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[52]  K. Hevner Experimental studies of the elements of expression in music , 1936 .

[53]  R. Plutchik The Nature of Emotions , 2001 .

[54]  Zhenqi Li,et al.  A Review of Emotion Recognition Using Physiological Signals , 2018, Sensors.

[55]  Yi-Hsuan Yang,et al.  Developing a benchmark for emotional analysis of music , 2017, PloS one.

[56]  Shashidhar G. Koolagudi,et al.  Content-Based Music Information Retrieval (CB-MIR) and Its Applications toward the Music Industry , 2018, ACM Comput. Surv..

[57]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[58]  L. Warrenburg Choosing the Right Tune , 2020 .

[59]  Nicu Sebe,et al.  AMIGOS: A dataset for Mood, personality and affect research on Individuals and GrOupS , 2017, ArXiv.

[60]  P. Lang The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[61]  Antonio Fernández-Caballero,et al.  Emotion Detection in Aging Adults Through Continuous Monitoring of Electro-Dermal Activity and Heart-Rate Variability , 2019, IWINAC.

[62]  Erik Wästlund,et al.  Let the music play...or not: the influence of background music on consumer behaviour , 2012 .

[63]  Zhouyu Fu,et al.  A Survey of Audio-Based Music Classification and Annotation , 2011, IEEE Transactions on Multimedia.

[64]  Remco C. Veltkamp,et al.  Studying emotion induced by music through a crowdsourcing game , 2016, Inf. Process. Manag..

[65]  Mohammad A. Khalilzadeh,et al.  EMOTIONAL STRESS RECOGNITION SYSTEM FOR AFFECTIVE COMPUTING BASED ON BIO-SIGNALS , 2010 .

[66]  Leontios J. Hadjileontiadis,et al.  Toward an EEG-Based Recognition of Music Liking Using Time-Frequency Analysis , 2012, IEEE Transactions on Biomedical Engineering.

[67]  Bao-Liang Lu,et al.  A multimodal approach to estimating vigilance using EEG and forehead EOG , 2016, Journal of neural engineering.

[68]  R. Thayer The biopsychology of mood and arousal , 1989 .

[69]  Syed Muhammad Anwar,et al.  Human emotion recognition and analysis in response to audio music using brain signals , 2016, Comput. Hum. Behav..

[70]  C. Fung,et al.  Benefits of Music Participation for Senior Citizens: A Review of the Literature , 2010 .

[71]  Bao-Liang Lu,et al.  Identifying Stable Patterns over Time for Emotion Recognition from EEG , 2016, IEEE Transactions on Affective Computing.

[72]  Yi-Hsuan Yang,et al.  A Regression Approach to Music Emotion Recognition , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[73]  W. David Hairston,et al.  EEG-Annotate: Automated identification and labeling of events in continuous signals with applications to EEG , 2018, Journal of Neuroscience Methods.

[74]  Christine L. Lisetti,et al.  Emotion recognition from physiological signals using wireless sensors for presence technologies , 2004, Cognition, Technology & Work.

[75]  Yasue Mitsukura,et al.  A feature extraction of the EEG during listening to the music using the factor analysis and neural networks , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[76]  Genesis R. Bojorquez,et al.  Music Therapy for Surgical Patients , 2020, Critical care nursing quarterly.

[77]  Sung-Bae Cho,et al.  A Context-Aware Music Recommendation System Using Fuzzy Bayesian Networks with Utility Theory , 2006, FSKD.

[78]  D. de Waard,et al.  The influence of music on mood and performance while driving , 2012, Ergonomics.

[79]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[80]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis , 2010, IEEE Transactions on Affective Computing.

[81]  Dae-Jin Kim,et al.  Analysis of Pre-Processing Methods for Music Information Retrieval in Noisy Environments using Mobile Devices , 2012 .