A Survey on Autonomous Techniques for Music Classification based on Human Emotions Recognition

Music is one of the finest element to trigger emotions in human beings. Each and every human being feels the music and emotions are automatically provoked by listening music. Music is considered as strong stress reliever. With the increase in size of music dataset available online and advancement of automation technologies the emotions from the music are to be recognized automatically so that the online database of music can be organized and browsed in an efficient manner. Automation of music emotion classification (MEC) helps the people to listen the music of their interest without wasting time on surfing the internet. It helps the psychologists in treatment process of patients. It also helps the musicians and artists to work on specific type of music and to classify them. This paper aims to provide the overview and survey related to autonomous technique for music classification (ATMC). In this article, the basic steps such as database collection, preprocessing, database analysis, feature extraction, classification and evaluation parameters involved in ATMC are explained and comprehensive review related to the basic steps is summarized. Research issues and solutions related to ATMC along with future scope are also discussed in this article.

[1]  C. C. Pratt,et al.  Music as a Language of Emotion , 1948 .

[2]  Yi-Hsuan Yang,et al.  Cross-Dataset and Cross-Cultural Music Mood Prediction: A Case on Western and Chinese Pop Songs , 2017, IEEE Transactions on Affective Computing.

[3]  Yongchuan Tang,et al.  Automatic Music Emotion Classification Using a New Classification Algorithm , 2009, 2009 Second International Symposium on Computational Intelligence and Design.

[4]  Aathreya S. Bhat,et al.  An Efficient Classification Algorithm for Music Mood Detection in Western and Hindi Music Using Audio Feature Extraction , 2014, 2014 Fifth International Conference on Signal and Image Processing.

[5]  Homer H. Chen,et al.  Music Emotion Recognition , 2011 .

[6]  Preeti Rao,et al.  VOCAL MELODY DETECTION IN THE PRESENCE OF PITCHED ACCOMPANIMENT USING HARMONIC MATCHING METHODS , 2008 .

[7]  Hardik Shah,et al.  Music Playlist Generation using Facial Expression Analysis and Task Extraction , 2017, Ann. UMCS Informatica.

[8]  Mert Bay,et al.  The 2007 MIREX Audio Mood Classification Task: Lessons Learned , 2008, ISMIR.

[9]  C. V. Jawahar,et al.  Multi-label annotation of music , 2015, 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR).

[10]  Constantine Kotropoulos,et al.  Automatic music mood classification via Low-Rank Representation , 2011, 2011 19th European Signal Processing Conference.

[11]  Jijun Wang,et al.  Emotional classification based on the tempo and mutation degrees , 2013, 2013 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation (IMSNA).

[12]  Bo Zhang,et al.  An Efficient Content Based Music Retrieval Algorithm , 2018, 2018 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS).

[13]  Benjamin Bischke,et al.  The Multimedia Satellite Task at MediaEval 2018: Emergency Response for Flooding Events , 2018 .

[14]  Yi-Hsuan Yang,et al.  Linear regression-based adaptation of music emotion recognition models for personalization , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Chang-Hong Lin,et al.  Happiness detection in music using hierarchical SVMs with dual types of kernels , 2013, 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.

[16]  Juan Li,et al.  Review of data features-based music emotion recognition methods , 2017, Multimedia Systems.

[17]  Junjie Bai,et al.  Music emotions recognition by cognitive classification methodologies , 2017, 2017 IEEE 16th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC).

[18]  Yi-Hsuan Yang,et al.  Modeling the Affective Content of Music with a Gaussian Mixture Model , 2015, IEEE Transactions on Affective Computing.

[19]  Fengxi Song,et al.  Feature Selection Using Principal Component Analysis , 2010, 2010 International Conference on System Science, Engineering Design and Manufacturing Informatization.

[20]  N. Chaikla,et al.  Genetic algorithms in feature selection , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[21]  Andreas F. Ehmann,et al.  Lyric Text Mining in Music Mood Classification , 2009, ISMIR.

[22]  Yueting Zhuang,et al.  Popular music retrieval by detecting mood , 2003, SIGIR.

[23]  Tomoko Matsui,et al.  Music Genre and Emotion Recognition Using Gaussian Processes , 2014, IEEE Access.

[24]  Bor-Shyh Lin,et al.  Emotion recognition of EEG underlying favourite music by support vector machine , 2013, 2013 1st International Conference on Orange Technologies (ICOT).

[25]  Sungho Jo,et al.  Deep Physiological Affect Network for the Recognition of Human Emotions , 2020, IEEE Transactions on Affective Computing.

[26]  Yi-Hsuan Yang,et al.  Ranking-Based Emotion Recognition for Music Organization and Retrieval , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[27]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[28]  R. Ross A statistic for circular series. , 1938 .

[29]  Jia-Ching Wang,et al.  Hierarchical Dirichlet Process Mixture Model for Music Emotion Recognition , 2015, IEEE Transactions on Affective Computing.

[30]  Björn Schuller,et al.  Opensmile: the munich versatile and fast open-source audio feature extractor , 2010, ACM Multimedia.

[31]  Thomas Fillon,et al.  YAAFE, an Easy to Use and Efficient Audio Feature Extraction Software , 2010, ISMIR.

[32]  Xavier Serra,et al.  Essentia: An Audio Analysis Library for Music Information Retrieval , 2013, ISMIR.

[33]  Gert R. G. Lanckriet,et al.  A Game-Based Approach for Collecting Semantic Annotations of Music , 2007, ISMIR.

[34]  Jose J. Valero-Mas,et al.  Interactive user correction of automatically detected onsets: approach and evaluation , 2017, EURASIP J. Audio Speech Music. Process..

[35]  Guojun Lu,et al.  Music emotion annotation by machine learning , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[36]  P. Wilson,et al.  The Nature of Emotions , 2012 .

[37]  Braja Gopal Patra,et al.  Multimodal mood classification of Hindi and Western songs , 2018, Journal of Intelligent Information Systems.

[38]  Eduardo Coutinho,et al.  Connecting Subspace Learning and Extreme Learning Machine in Speech Emotion Recognition , 2019, IEEE Transactions on Multimedia.

[39]  Yi-Hsuan Yang,et al.  A Regression Approach to Music Emotion Recognition , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[40]  Hossein Sameti,et al.  Music emotion recognition using two level classification , 2014, 2014 Iranian Conference on Intelligent Systems (ICIS).

[41]  Densil Cabrera,et al.  'Psysound3': Software for Acoustical and Psychoacoustical Analysis of Sound Recordings , 2007 .

[42]  Kjell Elenius,et al.  Emotion Recognition , 2009, Computers in the Human Interaction Loop.

[43]  Halina Kwasnicka,et al.  Music emotion analysis using semantic embedding recurrent neural networks , 2017, 2017 IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA).

[44]  K. Scherer,et al.  Emotions evoked by the sound of music: characterization, classification, and measurement. , 2008, Emotion.

[45]  Shrikanth Narayanan,et al.  Learning Shared Vector Representations of Lyrics and Chords in Music , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[46]  Tanaya Guha,et al.  Learning Affective Correspondence between Music and Image , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[47]  Gert R. G. Lanckriet,et al.  Semantic Annotation and Retrieval of Music and Sound Effects , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[48]  Damjan Vlaj,et al.  Acoustic classification and segmentation using modified spectral roll-off and variance-based features , 2013, Digit. Signal Process..

[49]  George Tzanetakis,et al.  MARSYAS: a framework for audio analysis , 1999, Organised Sound.

[50]  Ichiro Fujinaga,et al.  jAudio: An Feature Extraction Library , 2005, ISMIR.

[51]  J. Russell A circumplex model of affect. , 1980 .

[52]  Christian S. Jensen,et al.  Emotion-based music retrieval on a well-reduced audio feature space , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[53]  Yan Liu,et al.  CNN based music emotion classification , 2017, ArXiv.

[54]  Vahida Z. Attar,et al.  Mood classification of Indian popular music , 2012, CUBE.

[55]  Masayuki Numao,et al.  Subject-Independent Emotion Recognition During Music Listening Based on EEG Using Deep Convolutional Neural Networks , 2019, 2019 IEEE 15th International Colloquium on Signal Processing & Its Applications (CSPA).

[56]  Miki Haseyama,et al.  Novel Audio Feature Projection Using KDLPCCA-Based Correlation with EEG Features for Favorite Music Classification , 2019, IEEE Transactions on Affective Computing.

[57]  Fabien Ringeval,et al.  Leveraging Unlabeled Data for Emotion Recognition With Enhanced Collaborative Semi-Supervised Learning , 2018, IEEE Access.

[58]  O. Lartillot,et al.  A MATLAB TOOLBOX FOR MUSICAL FEATURE EXTRACTION FROM AUDIO , 2007 .

[59]  Preeti Rao,et al.  Singing Voice Detection in North Indian Classical Music , 2008 .

[60]  Donald S. Williamson,et al.  Towards Quantifying the "Album Effect" in Artist Identification , 2006, ISMIR.

[61]  Zhouyu Fu,et al.  A Survey of Audio-Based Music Classification and Annotation , 2011, IEEE Transactions on Multimedia.

[62]  Jacek Grekow,et al.  Audio features dedicated to the detection of arousal and valence in music recordings , 2017, 2017 IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA).

[63]  Juhan Nam,et al.  Multi-Level and Multi-Scale Feature Aggregation Using Sample-level Deep Convolutional Neural Networks for Music Classification , 2017, ICML 2017.

[64]  Junjie Bai,et al.  Dimensional music emotion recognition by valence-arousal regression , 2016, 2016 IEEE 15th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC).

[65]  Chuan-Yu Chang,et al.  Music emotion recognition with consideration of personal preference , 2011, The 2011 International Workshop on Multidimensional (nD) Systems.

[66]  P. Farnsworth A STUDY OF THE HEVNER ADJECTIVE LIST , 1954 .

[67]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[68]  Tuomas Eerola,et al.  Generalizability and Simplicity as Criteria in Feature Selection: Application to Mood Classification in Music , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[69]  Constantine Kotropoulos,et al.  Large scale musical instrument identification , 2007 .

[70]  Youngmoo E. Kim,et al.  Feature selection for content-based, time-varying musical emotion regression , 2010, MIR '10.

[71]  Anna Aljanaki Emotion in Music: representation and computational modeling , 2016 .

[72]  Juhan Nam,et al.  Multi-Level and Multi-Scale Feature Aggregation Using Pretrained Convolutional Neural Networks for Music Auto-Tagging , 2017, IEEE Signal Processing Letters.

[73]  Yu-Liang Hsu,et al.  Automatic ECG-Based Emotion Recognition in Music Listening , 2020, IEEE Transactions on Affective Computing.

[74]  Yuan-Pin Lin,et al.  EEG-Based Emotion Recognition in Music Listening , 2010, IEEE Transactions on Biomedical Engineering.

[75]  Seok-Pil Lee,et al.  Feature selection and comparison for the emotion recognition according to music listening , 2017, 2017 International Conference on Robotics and Automation Sciences (ICRAS).

[76]  In-Kwon Lee,et al.  Affecticon: Emotion-Based Icons for Music Retrieval , 2011, IEEE Computer Graphics and Applications.

[77]  K. MacDorman,et al.  Automatic Emotion Prediction of Song Excerpts: Index Construction, Algorithm Design, and Empirical Comparison , 2007 .

[78]  Hsin-Min Wang,et al.  A histogram density modeling approach to music emotion recognition , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[79]  Bob L. Sturm Evaluating music emotion recognition: Lessons from music genre recognition? , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[80]  Thierry Bertin-Mahieux,et al.  The Million Song Dataset , 2011, ISMIR.

[81]  Michael I. Mandel,et al.  Evaluation of Algorithms Using Games: The Case of Music Tagging , 2009, ISMIR.

[82]  J. Marozeau,et al.  Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts , 2005 .

[83]  Yi-Hsuan Yang,et al.  Prediction of the Distribution of Perceived Music Emotions Using Discrete Samples , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[84]  ChingShun Lin,et al.  Music emotion recognition based on two-level support vector classification , 2016, 2016 International Conference on Machine Learning and Cybernetics (ICMLC).

[85]  Sergio Canazza,et al.  Clustering Affective Qualities of Classical Music: Beyond the Valence-Arousal Plane , 2014, IEEE Transactions on Affective Computing.

[86]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[87]  Rui Pedro Paiva,et al.  Emotionally-Relevant Features for Classification and Regression of Music Lyrics , 2018, IEEE Transactions on Affective Computing.

[88]  Emery Schubert Modeling Perceived Emotion With Continuous Musical Features , 2004 .

[89]  Iwan Binanto A method of mood classification on keroncong music , 2018, 2018 IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE).

[90]  Fabien Ringeval,et al.  AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge , 2016, AVEC@ACM Multimedia.

[91]  Jianwei Niu,et al.  A Novel Method Based on OMPGW Method for Feature Extraction in Automatic Music Mood Classification , 2019, IEEE Transactions on Affective Computing.

[92]  Li Yang,et al.  Extraction and evaluation model for the basic characteristics of MIDI file music , 2014, The 26th Chinese Control and Decision Conference (2014 CCDC).

[93]  Tao Li,et al.  Detecting emotion in music , 2003, ISMIR.

[94]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[95]  Yi-Hsuan Yang,et al.  The AMG1608 dataset for music emotion recognition , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[96]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[97]  J. Stephen Downie,et al.  A framework for evaluating multimodal music mood classification , 2017, J. Assoc. Inf. Sci. Technol..

[98]  R. Thayer The biopsychology of mood and arousal , 1989 .

[99]  Alain Pruski,et al.  Human-Computer Interaction Using Emotion Recognition from Facial Expression , 2011, 2011 UKSim 5th European Symposium on Computer Modeling and Simulation.

[100]  Kasiprasad Mannepalli,et al.  Analysis of Emotion Recognition System for Telugu Using Prosodic and Formant Features , 2018 .

[101]  David Huron Perceptual and Cognitive Applications in Music Information Retrieval , 2000, ISMIR.

[102]  Eduardo Coutinho,et al.  Dynamic Difficulty Awareness Training for Continuous Emotion Prediction , 2018, IEEE Transactions on Multimedia.

[103]  Robert E. Davis,et al.  Statistics for the evaluation and comparison of models , 1985 .

[104]  Yi-Hsuan Yang,et al.  Towards time-varying music auto-tagging based on CAL500 expansion , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).