A review on sentiment discovery and analysis of educational big‐data

Sentiment discovery and analysis (SDA) aims to automatically identify the underlying attitudes, sentiments, and subjectivity towards a certain entity such as learners and learning resources. Due to its enormous potential for smart education, SDA has been deemed as a powerful technique for identifying and classifying sentiments from multimodal and multisource data over the whole process of education. For big educational data streams, SDA faces challenges in unimodal feature selection, sentiment classification, and multimodal fusion. As such, a large body of studies in the literature explores diverse approaches to SDA for educational applications. This paper provides a self‐contained, uniform overview of the SDA techniques for education. In particular, we focus on prominent studies in unimodal sentiment features and classifications (e.g., text, audio, and visual). In addition, we present a novel SDA framework of multimodal fusions, together with description of their crucial components. Based on this framework, we review different approaches to SDA on education from the perspectives of approaches and applications. After comprehensively reviewing the SDA techniques on education, we present the trends and prospectives of the future SDA research under ubiquitous education contexts.

[1]  Dianhui Wang,et al.  Stochastic Configuration Networks: Fundamentals and Algorithms , 2017, IEEE Transactions on Cybernetics.

[2]  Hua Xu,et al.  Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM , 2018, ArXiv.

[3]  Rosa M. Carro,et al.  Sentiment analysis in Facebook and its application to e-learning , 2014, Comput. Hum. Behav..

[4]  Ramón Zataraín-Cabada,et al.  Mining of Educational Opinions with Deep Learning , 2018, J. Univers. Comput. Sci..

[5]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[6]  Longzhuang Li,et al.  Recognizing and regulating e-learners' emotions based on interactive Chinese texts in e-learning systems , 2014, Knowl. Based Syst..

[7]  Deyu Li,et al.  A feature selection method based on improved fisher's discriminant ratio for text sentiment classification , 2011, Expert Syst. Appl..

[8]  Wanxiang Che,et al.  Sentence Compression for Aspect-Based Sentiment Analysis , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[9]  Eric P. Xing,et al.  Select-additive learning: Improving generalization in multimodal sentiment analysis , 2016, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[10]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[11]  Jie Tang,et al.  Can we understand van gogh's mood?: learning to infer affects from images in social networks , 2012, ACM Multimedia.

[12]  Pablo-César Muñoz-Carril,et al.  Computer-supported collaborative learning: An analysis of the relationship between interaction, emotional support and online collaborative tools , 2019, Comput. Educ..

[13]  Fabien Ringeval,et al.  Prediction-based learning for continuous emotion recognition in speech , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Yogesh Kumar Meena,et al.  Aspect-Based Sentiment Analysis of Students’ Feedback to Improve Teaching–Learning Process , 2018, Information and Communication Technology for Intelligent Systems.

[15]  Verónica Pérez-Rosas,et al.  Utterance-Level Multimodal Sentiment Analysis , 2013, ACL.

[16]  Li Li,et al.  Sentiment-enhanced learning model for online language learning system , 2018, Electron. Commer. Res..

[17]  David W. Jacobs,et al.  Generalized Multiview Analysis: A discriminative latent space , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  ThelwallMike,et al.  Sentiment strength detection in short informal text , 2010 .

[20]  Danielle S McNamara,et al.  Sentiment Analysis and Social Cognition Engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis , 2017, Behavior research methods.

[21]  Wen Gao,et al.  Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching , 2018, IEEE Transactions on Multimedia.

[22]  Bruno M. C. Silva,et al.  Towards a cooperative security system for mobile-health applications , 2014 .

[23]  Konstantinas Korovkinas,et al.  SVM and Naïve Bayes Classification Ensemble Method for Sentiment Analysis , 2017, Balt. J. Mod. Comput..

[24]  Cícero Nogueira dos Santos,et al.  Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts , 2014, COLING.

[25]  Prem Melville,et al.  Sentiment analysis of blogs by combining lexical knowledge with text classification , 2009, KDD.

[26]  Osmar R. Zaïane,et al.  Current State of Text Sentiment Analysis from Opinion to Emotion Mining , 2017, ACM Comput. Surv..

[27]  Orlando Troisi,et al.  Big data and sentiment analysis to highlight decision behaviours: a case study for student population , 2018, Behav. Inf. Technol..

[28]  Yunming Ye,et al.  Sentiment analysis through critic learning for optimizing convolutional neural networks with rules , 2019, Neurocomputing.

[29]  A. Kazak,et al.  Psychologists in Patient-Centered Medical Homes (PCMHs): Roles, Evidence, Opportunities, and Challenges , 2017, The American psychologist.

[30]  Erik Cambria,et al.  Sentiment Data Flow Analysis by Means of Dynamic Linguistic Patterns , 2015, IEEE Computational Intelligence Magazine.

[31]  Bert Arnrich,et al.  Towards an Emotional Engagement Model: Can Affective States of a Learner be Automatically Detected in a 1: 1 Learning Scenario? , 2016, UMAP.

[32]  Turgut Özseven,et al.  Investigation of the effect of spectrogram images and different texture analysis methods on speech emotion recognition , 2018, Applied Acoustics.

[33]  Erik Cambria,et al.  Multimodal Sentiment Analysis: Addressing Key Issues and Setting Up the Baselines , 2018, IEEE Intelligent Systems.

[34]  A. Graesser,et al.  Dynamics of affective states during complex learning , 2012 .

[35]  Yang Wang,et al.  Facial Feature Tracking using a Multi-State Hierarchical Shape Model under Varying Face Pose and Facial Expression , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[36]  G. Ram Mohana Reddy,et al.  An E-Learning System with Multifacial Emotion Recognition Using Supervised Machine Learning , 2015, 2015 IEEE Seventh International Conference on Technology for Education (T4E).

[37]  Michael W. Kraus Voice-Only Communication Enhances Empathic Accuracy , 2017, The American psychologist.

[38]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.

[39]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Erik Cambria,et al.  Affective Computing and Sentiment Analysis , 2016, IEEE Intelligent Systems.

[41]  Abdolhossein Sarrafzadeh,et al.  "How do you know that I don't understand?" A look at the future of intelligent tutoring systems , 2008, Comput. Hum. Behav..

[42]  Sen Wang,et al.  Multimodal sentiment analysis with word-level fusion and reinforcement learning , 2017, ICMI.

[43]  Daniel A. Keim,et al.  Visual sentiment analysis of customer feedback streams using geo-temporal term associations , 2013, Inf. Vis..

[44]  Dianhui Wang,et al.  Randomness in neural networks: an overview , 2017, WIREs Data Mining Knowl. Discov..

[45]  Erik Cambria,et al.  Fusing audio, visual and textual clues for sentiment analysis from multimodal content , 2016, Neurocomputing.

[46]  Yueh-Min Huang,et al.  Using Facial Expression to Detect Emotion in E-learning System: A Deep Learning Method , 2017, SETE@ICWL.

[47]  Joanne Yoo,et al.  Teacher emotion and learning as praxis: Professional development that matters , 2017 .

[48]  Azin Khosravi,et al.  The Iranian Vital Horoscope; Appropriate Tool to Collect Health Statistics in Rural Areas , 2009 .

[49]  Marcelo Worsley,et al.  Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks , 2016, J. Learn. Anal..

[50]  Erik Cambria,et al.  Deep Convolutional Neural Network Textual Features and Multiple Kernel Learning for Utterance-level Multimodal Sentiment Analysis , 2015, EMNLP.

[51]  Murtaza Haider,et al.  Beyond the hype: Big data concepts, methods, and analytics , 2015, Int. J. Inf. Manag..

[52]  Takeaki Uno,et al.  Student mental state inference from unintentional body gestures using dynamic Bayesian networks , 2010, Journal on Multimodal User Interfaces.

[53]  Bing Li,et al.  Context-aware affective images classification based on bilayer sparse representation , 2012, ACM Multimedia.

[54]  Takeaki Uno,et al.  Probabilistic Prediction of Student Affect from Hand Gestures , 2008, ARCS.

[55]  Yanning Zhang,et al.  Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[56]  Quratulain Rajput,et al.  Lexicon-Based Sentiment Analysis of Teachers' Evaluation , 2016, Appl. Comput. Intell. Soft Comput..

[57]  Yue Gao,et al.  Exploring Principles-of-Art Features For Image Emotion Recognition , 2014, ACM Multimedia.

[58]  Lun-Wei Ku,et al.  EmotionLines: An Emotion Corpus of Multi-Party Conversations , 2018, LREC.

[59]  Wil M. P. van der Aalst,et al.  Business Process Variability Modeling , 2017, ACM Comput. Surv..

[60]  Jiebo Luo,et al.  Aesthetics and Emotions in Images , 2011, IEEE Signal Processing Magazine.

[61]  Sergio Salmeron-Majadas,et al.  Emotions Detection from Math Exercises by Combining Several Data Sources , 2013, AIED.

[62]  Rada Mihalcea,et al.  MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations , 2018, ACL.

[63]  Loïc Kessous,et al.  Multimodal emotion recognition from expressive faces, body gestures and speech , 2007, AIAI.

[64]  Eric P. Xing,et al.  Select-Additive Learning: Improving Cross-individual Generalization in Multimodal Sentiment Analysis , 2016, ArXiv.

[65]  Neil Cooke,et al.  Affect recognition in an interactive gaming environment using eye tracking , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[66]  Angel Jannasch-Pennell,et al.  Enhancement of Student Experience Management in Higher Education by Sentiment Analysis and Text Mining , 2018, Int. J. Technol. Educ. Mark..

[67]  Russell Beale,et al.  Affective interaction: How emotional agents affect users , 2009, Int. J. Hum. Comput. Stud..

[68]  P. Ekman An argument for basic emotions , 1992 .

[69]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[70]  Jonathon S. Hare,et al.  Analyzing and predicting sentiment of images on the social web , 2010, ACM Multimedia.

[71]  Pierre Dumouchel,et al.  Anchor Models for Emotion Recognition from Speech , 2013, IEEE Transactions on Affective Computing.

[72]  Albino Nogueiras,et al.  Speech emotion recognition using hidden Markov models , 2001, INTERSPEECH.

[73]  Björn W. Schuller,et al.  YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context , 2013, IEEE Intelligent Systems.

[74]  John H. L. Hansen,et al.  Automatic audio sentiment extraction using keyword spotting , 2015, INTERSPEECH.

[75]  Walaa Medhat,et al.  Sentiment analysis algorithms and applications: A survey , 2014 .

[76]  Roger Levy,et al.  A new approach to cross-modal multimedia retrieval , 2010, ACM Multimedia.

[77]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  S. V. N. Vishwanathan,et al.  SPF-GMKL: generalized multiple kernel learning with a million kernels , 2012, KDD.

[79]  Reinhard Pekrun,et al.  Emotion Transmission in the Classroom Revisited: A Reciprocal Effects Model of Teacher and Student Enjoyment , 2017, Journal of Educational Psychology.

[80]  Erik Cambria,et al.  A review of affective computing: From unimodal analysis to multimodal fusion , 2017, Inf. Fusion.

[81]  Benjamin J. Southwell,et al.  Human Object Recognition Using Colour and Depth Information from an RGB-D Kinect Sensor , 2013 .

[82]  Erik Cambria,et al.  Combining Textual Clues with Audio-Visual Information for Multimodal Sentiment Analysis , 2018 .

[83]  Jillian Yelinek,et al.  ‘Show me your mad faces!’ preschool teachers’ emotion talk in the classroom , 2019 .

[84]  Seong-Joo Kim,et al.  Evolvable Recommendation System in the Portable Device Based on the Emotion Awareness , 2005, KES.

[85]  Paul L. Rosin,et al.  Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions , 2018, IEEE Transactions on Multimedia.

[86]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[87]  J.-M. Sun,et al.  Facial emotion recognition in modern distant education system using SVM , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[88]  Subhasmita Sahoo,et al.  Study of feature combination using HMM and SVM for multilingual Odiya speech emotion recognition , 2015, International Journal of Speech Technology.

[89]  Carina S. González-González,et al.  Facial Emotion Analysis in Down's syndrome children in classroom , 2015, Interacción.

[90]  Huosong Xia,et al.  Sentiment analysis for online reviews using conditional random fields and support vector machines , 2020, Electron. Commer. Res..

[91]  Paolo Rosso,et al.  Making objective decisions from subjective data: Detecting irony in customer reviews , 2012, Decis. Support Syst..

[92]  Dong Yu,et al.  Speech emotion recognition using deep neural network and extreme learning machine , 2014, INTERSPEECH.

[93]  P. Adinolfi,et al.  Perceived Innovative Teaching Procedures in Higher Education From Students' Perspectives From a Sentiment Analysis Approach , 2018 .

[94]  Fangzhao Wu,et al.  Microblog sentiment classification with heterogeneous sentiment knowledge , 2016, Inf. Sci..

[95]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[96]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[97]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[98]  Jatinderkumar R. Saini,et al.  Emotion Detection and Sentiment Analysis in Text Corpus: A Differential Study with Informal and Formal Writing Styles , 2014 .

[99]  Mohammad Soleymani,et al.  A survey of multimodal sentiment analysis , 2017, Image Vis. Comput..

[100]  Mohd Sharifuddin Ahmad,et al.  An emotion-based model for improving students' engagement using agent-based social simulator , 2016 .

[101]  Anne C. Frenzel,et al.  Measuring emotions in students learning and performance: The Achievement Emotions Questionnaire (AE , 2011 .

[102]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[103]  Chih-Hung Wu,et al.  Review of affective computing in education/learning: Trends and challenges , 2016, Br. J. Educ. Technol..

[104]  Lori Lockyer,et al.  Seeking optimal confusion: a review on epistemic emotion management in interactive digital learning environments , 2019, Interact. Learn. Environ..

[105]  Tao Xu,et al.  Learning Emotions EEG-based Recognition and Brain Activity: A Survey Study on BCI for Intelligent Tutoring System , 2018, ANT/SEIT.

[106]  J. M. Martin,et al.  SentBuk: Sentiment analysis for e-learning environments , 2012, 2012 International Symposium on Computers in Education (SIIE).

[107]  Wioleta Szwoch,et al.  Emotion Recognition Using Physiological Signals , 2015, MIDI '15.

[108]  Souraya Ezzat,et al.  Sentiment Analysis of Call Centre Audio Conversations using Text Classification , 2022 .

[109]  Alex Waibel,et al.  Detecting Emotions in Speech , 1998 .

[110]  Kiavash Bahreini,et al.  Towards real-time speech emotion recognition for affective e-learning , 2015, Education and Information Technologies.

[111]  Raymond Y. K. Lau,et al.  Big data analytics for security and criminal investigations , 2017, WIREs Data Mining Knowl. Discov..

[112]  Lei Zhang,et al.  Sentiment Analysis and Opinion Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[113]  Pilsung Kang,et al.  Sentiment Classification with Word Attention based on Weakly Supervised Learning with a Convolutional Neural Network , 2017, ArXiv.

[114]  Xiaodan Huang,et al.  Physiological Signals based Day-Dependence Analysis with Metric Multidimensional Scaling for Sentiment Classification in Wearable Sensors , 2015 .

[115]  Ting Liu,et al.  Deep learning for sentiment analysis: successful approaches and future challenges , 2015, WIREs Data Mining Knowl. Discov..

[116]  Shafaatunnur Hasan,et al.  Machine learning-based multi-documents sentiment-oriented summarization using linguistic treatment , 2018, Expert Syst. Appl..

[117]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[118]  Peter F. Driessen,et al.  Gesture-Based Affective Computing on Motion Capture Data , 2005, ACII.

[119]  Justin Zhijun Zhan,et al.  Sentiment analysis using product review data , 2015, Journal of Big Data.

[120]  Björn W. Schuller,et al.  Sentiment analysis and opinion mining: on optimal parameters and performances , 2015, WIREs Data Mining Knowl. Discov..

[121]  Angeliki Metallinou,et al.  Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice , 2008, 2008 Tenth IEEE International Symposium on Multimedia.

[122]  Taghi M. Khoshgoftaar,et al.  Deep learning applications and challenges in big data analytics , 2015, Journal of Big Data.

[123]  Lia M. Daniels,et al.  Not that different in theory: Discussing the control-value theory of emotions in online learning environments , 2012, Internet High. Educ..

[124]  S. Connelly,et al.  Negative emotions in informal feedback: The benefits of disappointment and drawbacks of anger , 2014 .

[125]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[126]  Tong Lu,et al.  A Novel Multi-modal Integration and Propagation Model for Cross-Media Information Retrieval , 2012, MMM.

[127]  Erik Cambria,et al.  Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[128]  Jiebo Luo,et al.  Sentribute: image sentiment analysis from a mid-level perspective , 2013, WISDOM '13.

[129]  M. D. Meijer The contribution of general features of body movement to the attribution of emotions , 1989 .

[130]  Peter D. Turney Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[131]  Louis-Philippe Morency,et al.  Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages , 2016, IEEE Intelligent Systems.

[132]  Diego Reforgiato Recupero,et al.  Frame-Based Detection of Opinion Holders and Topics: A Model and a Tool , 2014, IEEE Computational Intelligence Magazine.

[133]  Juan-Zi Li,et al.  How Do Your Friends on Social Media Disclose Your Emotions? , 2014, AAAI.

[134]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[135]  T. Chao,et al.  Sitagliptin and the risk of hospitalization for heart failure: a population-based study. , 2014, International journal of cardiology.

[136]  Bin Hu,et al.  Towards affective learning with an EEG feedback approach , 2009, MTDL '09.

[137]  Vitomir Štruc,et al.  Towards Efficient Multi-Modal Emotion Recognition , 2013 .

[138]  Jiebo Luo,et al.  Visual Sentiment Analysis by Attending on Local Image Regions , 2017, AAAI.