Understanding the effects of real-time sentiment analysis and morale visualisation in backchannel systems: A case study

Abstract When presenting to a large group of students, either in an amphitheatre or through an online platform, effectively connecting to the audience – understanding how well the audience is following the presentation and taking appropriate actions promptly if they experience difficulties – is a serious challenge. Backchannel systems are sometimes deployed to address this issue by allowing audience to give feedback to the presenter without interrupting the current discourse. However, these systems are not designed to immediately aggregate and present the audience's feedback to the presenter in a meaningful way that is easy to quickly digest. To fill this gap, we have explored a proof-of-concept method for analysing the emotions and sentiments from the audience's feedback in real time and displaying to the presenter a morale graph showing a trend of the audience's overall reaction over time. This allows a presenter to effectively connected to their audience in real time, knowing whether their presentation is going well and what issues their audience may have in common at any specific moment. We have further implemented this method in an educational context, using a prototype backchannel system, known as ClasSense, for a lecturer to effectively connect to their students. This paper presents the evaluation of this system, which shows that lecturers accept and prefer the morale graph based user interface developed over other backchannel user interfaces that display all posts in chronological order. Students also positively expressed their agreement that the system not only makes their feedback an important part of the class but also increases their interactions with the lecturers. This is further confirmed with a Markov chain predicting the probability that students’ and lecturers’ survey results lead to their overall positive sentiment towards the tool. The flexibility of the ClasSense system suggests it may also be suitable in contexts other than education.

[1]  S. M. Ferdous Azam,et al.  Academicians' Acceptance of Online Learning Environments: A Review of Information System Theories and Models , 2019, Global Journal of Computer Science and Technology.

[2]  Reynol Junco,et al.  The effect of Twitter on college student engagement and grades , 2011, J. Comput. Assist. Learn..

[3]  Krishna Bista,et al.  Asian International Students’ College Experience: Relationship between Quality of Personal Contact and Gains in Learning , 2015 .

[4]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[5]  Hiroya Takamura,et al.  Sentiment Classification Using Word Sub-sequences and Dependency Sub-trees , 2005, PAKDD.

[6]  Rob Reilly,et al.  Analytical Models of Emotions , Learning and Relationships : Towards an Affect-sensitive Cognitive Machine , 2001 .

[7]  Robert Goodwin,et al.  ClasSense: A Mobile Digital Backchannel System for Monitoring Class Morale , 2015 .

[8]  Kenneth Y. Goldberg,et al.  Opinion space: a scalable tool for browsing online comments , 2010, CHI.

[9]  Steven M. Oberhelman,et al.  Globally Networked Learning in a University Classroom: A Pilot Program , 2019, Athens Journal of Education.

[10]  Erkki Sutinen,et al.  Exploiting sentiment analysis to track emotions in students' learning diaries , 2013, Koli Calling '13.

[11]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[12]  S. Bennett,et al.  Technological diversity: an investigation of students’ technology use in everyday life and academic study , 2010 .

[13]  Rosa M. Carro,et al.  Sentiment analysis in Facebook and its application to e-learning , 2014, Comput. Hum. Behav..

[14]  Jules White,et al.  Smartphone Computing in the Classroom , 2011, IEEE Pervasive Computing.

[15]  Michael S. Bernstein,et al.  Twitinfo: aggregating and visualizing microblogs for event exploration , 2011, CHI.

[16]  Andreas M. Kaplan,et al.  The early bird catches the news: Nine things you should know about micro-blogging , 2011 .

[17]  Larisa Olesova,et al.  Hotseat: Opening the Backchannel in Large Lectures , 2010 .

[18]  Tatsuya Kawahara,et al.  Prediction and Generation of Backchannel Form for Attentive Listening Systems , 2016, INTERSPEECH.

[19]  J. Russell.  Core affect and the psychological construction of emotion. , 2003, Psychological review.

[20]  Ann Henderson,et al.  The Seven Principles of Good Practice: Applications for Online Education in Nursing , 2002, Nurse educator.

[21]  Sheizaf Rafaeli,et al.  Networked Learning Analytics: A Theoretically Informed Methodology for Analytics of Collaborative Learning , 2019, Learning In a Networked Society.

[22]  Frank Pajares,et al.  Gender and Perceived Self-Efficacy in Self-Regulated Learning , 2002 .

[23]  Gwen Littlewort,et al.  Automatic Recognition of Facial Actions in Spontaneous Expressions , 2006, J. Multim..

[24]  D. Schunk.  Ability Versus Effort Attributional Feedback: Differential Effects on Self-Efficacy and Achievement , 1983 .

[25]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[26]  A. Artino.  Online or face-to-face learning? Exploring the personal factors that predict students' choice of instructional format , 2010, Internet High. Educ..

[27]  Lori Lockyer,et al.  Identifying epistemic emotions from activity analytics in interactive digital learning environments , 2018, Learning Analytics in the Classroom.

[28]  Marc A. Brackett,et al.  Classroom emotional climate, student engagement, and academic achievement , 2012 .

[29]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[30]  I. Ajzen,et al.  Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research , 1977 .

[31]  Hal Abelson,et al.  The Creation of OpenCourseWare at MIT , 2008 .

[32]  Michael Gamon,et al.  Sentiment classification on customer feedback data: noisy data, large feature vectors, and the role of linguistic analysis , 2004, COLING.

[33]  Choochart Haruechaiyasak,et al.  Discovering Consumer Insight from Twitter via Sentiment Analysis , 2012, J. Univers. Comput. Sci..

[34]  Marilla D. Svinicki,et al.  New Directions in Learning and Motivation. , 1999 .

[35]  Fred D. Davis.  User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[36]  Rob Malouf,et al.  A Preliminary Investigation into Sentiment Analysis of Informal Political Discourse , 2006, AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs.

[37]  Tai-Hung Lee,et al.  Emotion recognition and communication for reducing second‐language speaking anxiety in a web‐based one‐to‐one synchronous learning environment , 2011 .

[38]  Mar Pérez-Sanagustín,et al.  Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses , 2017, Comput. Educ..

[39]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[40]  Vicky G. Spencer,et al.  Experiences of instructors in online learning environments: Identifying and regulating emotions , 2012, Internet High. Educ..

[41]  Fred Paas,et al.  Supporting Self-Regulated Learning in Online Learning Environments and MOOCs: A Systematic Review , 2018, Int. J. Hum. Comput. Interact..

[42]  Phyllis Jones,et al.  Virtual Spaces: Employing a Synchronous Online Classroom to Facilitate Student Engagement in Online Learning , 2009 .

[43]  Emma Mayhew,et al.  No Longer a Silent Partner: How Mentimeter Can Enhance Teaching and Learning Within Political Science , 2019, Journal of Political Science Education.

[44]  Yibin Li,et al.  Impact of using classroom response systems on students' entrepreneurship learning experience , 2017, Comput. Hum. Behav..

[45]  Prem Melville,et al.  Sentiment analysis of blogs by combining lexical knowledge with text classification , 2009, KDD.

[46]  Nina Bonderup Dohn,et al.  Is Networked Learning Postdigital Education? , 2019, Postdigital Science and Education.

[47]  Qinghua Zheng,et al.  Can e-Learner's emotion be recognized from interactive Chinese texts? , 2009, 2009 13th International Conference on Computer Supported Cooperative Work in Design.

[48]  D. Sadler.  Beyond feedback: developing student capability in complex appraisal , 2010 .

[49]  Moshe Zviran,et al.  User satisfaction from commercial web sites: The effect of design and use , 2006, Inf. Manag..

[50]  Walter Brenner,et al.  Massive Open Online Courses , 2014 .

[51]  Jill A. Marshall,et al.  Classroom Response Systems: A Review of the Literature , 2006 .

[52]  David M. Pennock,et al.  Mining the peanut gallery: opinion extraction and semantic classification of product reviews , 2003, WWW '03.

[53]  Thomas Ryberg,et al.  Celebrating the Tenth Networked Learning Conference: Looking Back and Moving Forward , 2018 .

[54]  Andrea Esuli,et al.  SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining , 2010, LREC.

[55]  Stefan Hrastinski,et al.  Asynchronous & Synchronous E-Learning , 2008 .

[56]  Peter D. Turney.  Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews , 2002, ACL.

[57]  G. DeBourgh,et al.  Use of classroom "clickers" to promote acquisition of advanced reasoning skills. , 2008, Nurse education in practice.

[58]  Mary Beth Rosson,et al.  Augmenting classroom participation through public digital backchannels , 2012, GROUP.

[59]  Lawrence B. Wolff,et al.  Tracking human faces in infrared video , 2003, Image Vis. Comput..

[60]  Haifeng Shen,et al.  Sentiment analysis and visualisation in a backchannel system , 2016, OZCHI.

[61]  J. Arbaugh,et al.  Technological and Structural Characteristics, Student Learning and Satisfaction with Web-Based Courses , 2002 .

[62]  Ava G Porter,et al.  Evaluating the effect of interactive audience response systems on the perceived learning experience of nursing students. , 2010, The Journal of nursing education.

[63]  Keenan A. Pituch,et al.  The influence of system characteristics on e-learning use , 2006, Comput. Educ..

[64]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[65]  Michaela Gläser-Zikuda,et al.  Promoting Students' Emotions and Achievement--Instructional Design and Evaluation of the ECOLE-Approach. , 2005 .

[66]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[67]  E. Gagné.  The cognitive psychology of school learning , 1985 .

[68]  Jean-Yves Antoine,et al.  Weighted Krippendorff’s alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation , 2014, EACL.

[69]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[70]  Barbara J. Millis,et al.  Cooperative Learning for Higher Education Faculty , 1997 .

[71]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[72]  Anthony R. Artino,et al.  Emotions in online learning environments: Introduction to the special issue , 2012, Internet High. Educ..

[73]  Edoardo M. Airoldi,et al.  Markov Blankets and Meta-heuristics Search: Sentiment Extraction from Unstructured Texts , 2004, WebKDD.

[74]  Chen Qiao,et al.  What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach , 2020, Comput. Educ..

[75]  Timothy J. Hickey,et al.  The affective tutor , 2014 .

[76]  Arvid Kappas,et al.  Sentiment in short strength detection informal text , 2010, J. Assoc. Inf. Sci. Technol..

[77]  H. Shachar,et al.  Talking, Relating, and Achieving: Effects of Cooperative Learning and Whole-Class Instruction , 1994 .

[78]  Phyllis C. Blumenfeld,et al.  Predicting Achievement Early and Late in the Semester: The Role of Motivation and Use of Learning Strategies. , 1990 .

[79]  Daryl J. D'Souza,et al.  Facebook versus Blackboard for Supporting the Learning of Programming in a Fully Online Course: The Changing Face of Computing Education , 2013, 2013 Learning and Teaching in Computing and Engineering.

[80]  Kuo-An Hwang,et al.  Assessment of affective state in distance learning based on image detection by using fuzzy fusion , 2009, Knowl. Based Syst..

[81]  A. Chickering,et al.  Seven Principles for Good Practice in Undergraduate Education , 1987, CORE.

[82]  Ard W. Lazonder,et al.  Modelling human emotions for tactical decision-making games , 2013, Br. J. Educ. Technol..

[83]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[84]  Ling He,et al.  Effects of social-interactive engagement on the dropout ratio in online learning: insights from MOOC , 2018, Behav. Inf. Technol..

[85]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[86]  Ben Shneiderman,et al.  Monitoring Academic Conferences: Real-Time Visualization and Retrospective Analysis of Backchannel Conversations , 2012, 2012 International Conference on Social Informatics.

[87]  Ashley Ann Skylar,et al.  A Comparison of Asynchronous Online Text-Based Lectures and Synchronous Interactive Web Conferencing Lectures , 2009 .

[88]  Richard J. Anderson,et al.  Promoting Interaction in Large Classes with Computer-Mediated Feedback , 2003, CSCL.

[89]  Sarita Yardi.  The role of the backchannel in collaborative learning environments , 2006 .

[90]  P. Schutz,et al.  Inquiry on Emotions in Education , 2002 .

[91]  T. Goetz,et al.  Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research , 2002 .

[92]  Anson Wong.  Classroom Response Systems and Student Performance Improvement: Local Versus International Students , 2016 .

[93]  Qing Chen,et al.  VisMOOC: Visualizing video clickstream data from massive open online courses , 2015, 2014 IEEE Conference on Visual Analytics Science and Technology (VAST).

[94]  Jeff Sauro,et al.  When designing usability questionnaires, does it hurt to be positive? , 2011, CHI.

[95]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[96]  Xiaojun Wan,et al.  Emotion Classification in Microblog Texts Using Class Sequential Rules , 2014, AAAI.

[97]  Joseph P. Mazer,et al.  Emotion in Teaching and Learning: Development and Validation of the Classroom Emotions Scale , 2010 .

[98]  Hong Yu,et al.  Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences , 2003, EMNLP.

[99]  Charles R. Graham,et al.  Empowering or compelling reluctant participators using audience response systems , 2007 .

[100]  Thomas A. Brush,et al.  Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors , 2008, Comput. Educ..

[101]  N. Morgan,et al.  Teaching, Questioning and Learning , 1991 .

[102]  Andy P. Field,et al.  Discovering Statistics Using Ibm Spss Statistics , 2017 .

[103]  James A. Elwood,et al.  Demotivation: Affective states and learning outcomes , 2009 .

[104]  Minseok Kang,et al.  Structural Relationships of Factors Which Impact on Learner Achievement in Online Learning Environment , 2019, The International Review of Research in Open and Distributed Learning.

[105]  Viswanath Venkatesh,et al.  Technology Acceptance Model 3 and a Research Agenda on Interventions , 2008, Decis. Sci..

[106]  Matthew Lombard.  A Call for Standardization in Content Analysis Reliability , 2004 .

[107]  Shonali Krishnaswamy,et al.  RnR: A System for Extracting Rationale from Online Reviews and Ratings , 2010, ICSOC.

[108]  Pilar Rodríguez Marín,et al.  Extracting Emotions from Texts in E-Learning Environments , 2012, 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems.

[109]  David A. Clifton,et al.  Detecting Adolescent Psychological Pressures from Micro-Blog , 2014, HIS.

[110]  Robert W. Lent,et al.  Relation of self-efficacy expectations to academic achievement and persistence. , 1984 .

[111]  Kumar Laxman,et al.  A study on the adoption of clickers in higher education , 2011 .

[112]  François Bry,et al.  Introducing Backstage - a digital backchannel for large class lectures , 2011, Interact. Technol. Smart Educ..

[113]  R. Weaver,et al.  Classroom Organization and Participation: College Students' Perceptions , 2005 .

[114]  David Baron,et al.  Investigating the effects of a backchannel on university classroom interactions: A mixed-method case study , 2016, Comput. Educ..

[115]  Matthew L. Meuter,et al.  A Student View of Technology in the Classroom , 2011 .

[116]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[117]  Lorenzo Magnani,et al.  Model-Based Reasoning in Scientific Discovery , 1999, Springer US.

[118]  Navneet Kaur,et al.  Opinion mining and sentiment analysis , 2016, 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom).

[119]  Jeffrey T. Hancock,et al.  Expressing emotion in text-based communication , 2007, CHI.

[120]  Gilly Salmon,et al.  The space for social media in structured online learning , 2015 .

[121]  Carolyn Penstein Rosé,et al.  Sentiment Analysis in MOOC Discussion Forums: What does it tell us? , 2014, EDM.

[122]  Stefan Hrastinski,et al.  Asynchronous and synchronous e-learning , 2008 .

[123]  A. Bangert.  The Seven Principles of Good Practice: A framework for evaluating on-line teaching , 2004, Internet High. Educ..

[124]  Wolff-Michael Roth,et al.  Differential Participation During Science Conversations: The Interaction of Display Artifacts, Social Configurations, and Physical Arrangements , 1996, ICLS.

[125]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[126]  Laura Czerniewicz,et al.  The Unbundled University: Researching Emerging Models in an Unequal Landscape , 2018 .

[127]  Scott R. Homan,et al.  Student evaluation of audience response technology in large lecture classes , 2008 .

[128]  Carlo Strapparava,et al.  SemEval-2007 Task 14: Affective Text , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[129]  Lorna Suen.  Teaching epidemiology using WebCT: application of the seven principles of good practice. , 2005, The Journal of nursing education.

[130]  Cheolil Lim,et al.  Design principles for improving emotional affordances in an online learning environment , 2018, Asia Pacific Education Review.

[131]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[132]  Mor Naaman,et al.  Diamonds in the rough: Social media visual analytics for journalistic inquiry , 2010, 2010 IEEE Symposium on Visual Analytics Science and Technology.

[133]  Janyce Wiebe,et al.  Development and Use of a Gold-Standard Data Set for Subjectivity Classifications , 1999, ACL.

[134]  J. Arbaugh.  Managing the on-line classroom , 2002 .

[135]  N. Hara.  STUDENT DISTRESS IN A WEB-BASED DISTANCE EDUCATION COURSE , 2000 .

[136]  Shonali Krishnaswamy,et al.  RnR: Extracting Rationale from Online Reviews and Ratings , 2010, 2010 IEEE International Conference on Data Mining Workshops.

[137]  Norliza Katuk,et al.  ASSESSMENT OF STUDENTS’ COGNITIVE–AFFECTIVE STATES IN LEARNING WITHIN A COMPUTER-BASED ENVIRONMENT: EFFECTS ON PERFORMANCE , 2015, Journal of Information and Communication Technology.

[138]  Armando Fox,et al.  Monitoring MOOCs: which information sources do instructors value? , 2014, L@S.

[139]  Timothy Pelton,et al.  15. Clicker Lessons: Assessing and Addressing Student Responses to Audience Response Systems , 2011 .

[140]  M. Ali Akber Dewan,et al.  Engagement detection in online learning: a review , 2019, Smart Learning Environments.

[141]  James R. Lewis.  Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .

[142]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[143]  Allison Littlejohn,et al.  Learning in MOOCs: Motivations and self-regulated learning in MOOCs , 2016, Internet High. Educ..

[144]  Martha Cleveland Innes,et al.  Emotional presence, learning, and the online learning environment , 2012 .

[145]  Lisa Linnenbrink-Garcia,et al.  Academic Emotions and Student Engagement , 2012 .

[146]  Dale S. Niederhauser,et al.  Promoting Deep and Durable Learning in the Online Classroom , 2000 .

[147]  Vladan Devedzic,et al.  Visualizing the Affective Structure of Students Interaction , 2012, ICHL.

[148]  Teoh Kung-Keat,et al.  Confused, Bored, Excited? An Emotion Based Approach to the Design of Online Learning Systems , 2016 .

[149]  Vladan Devedzic,et al.  Synesketch: An Open Source Library for Sentence-Based Emotion Recognition , 2013, IEEE Transactions on Affective Computing.

[150]  Stephen Ko,et al.  Interactivity, active collaborative learning, and learning performance: The moderating role of perceived fun by using personal response systems , 2019, The International Journal of Management Education.

[151]  Timothy W. Finin,et al.  Why we twitter: understanding microblogging usage and communities , 2007, WebKDD/SNA-KDD '07.

[152]  C. White,et al.  Higher education emotions: a scale development exercise , 2013 .

[153]  E. Wagner.  Enabling Mobile Learning. , 2005 .

[154]  Marian G. Williams,et al.  Backchannel: whispering in digital conversation , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[155]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[156]  Andrew McCallum,et al.  Using Maximum Entropy for Text Classification , 1999 .

[157]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[158]  Claire Cardie,et al.  Annotating Expressions of Opinions and Emotions in Language , 2005, Lang. Resour. Evaluation.

[159]  Arthur C. Graesser,et al.  SMART Environments That Support Monitoring, Reflection, and Revision , 1998 .