Integrating Learning Analytics to Measure Message Quality in Large Online Conversations

Research on computer-supported collaborative learning often employs content analysis as an approach to investigate message quality in asynchronous online discussions using systematic message-coding schemas. Although this approach helps researchers count the frequencies by which students engage in different socio-cognitive actions, it does not explain how students articulate their ideas in categorized messages. This study investigates the effects of a recommender system on the quality of students’ messages from voluminous discussions. We employ learning analytics to produce a quasi-quality index score for each message. Moreover, we examine the relationship between this score and the phases of a popular message-coding schema. Empirical findings show that a custom CSCL environment extended by a recommender system supports students to explore different viewpoints and modify interpretations with higher quasi-quality index scores than students assigned to the control software. Theoretical and practical implications are also discussed.

[1]  Marlena J. Gaul.  Big Data at Work: Dispelling the Myths, Uncovering the Opportunities , 2014 .

[2]  David H. Olsen,et al.  An exploratory study on issues and challenges of agile software development with scrum , 2008 .

[3]  Camillia Matuk,et al.  Why and how do middle school students exchange ideas during science inquiry? , 2018, Int. J. Comput. Support. Collab. Learn..

[4]  Neil Mercer,et al.  Words and Minds : How We Use Language to Think Together , 2000 .

[5]  Anastasios A. Economides,et al.  Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence , 2014, J. Educ. Technol. Soc..

[6]  Tammy Schellens,et al.  Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review , 2006, Comput. Educ..

[7]  Terry Ryan,et al.  The Design and Evaluation of a Peer Ratings System for Online Learning Communities , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[8]  Ming Ming Chiu,et al.  Design and evaluation of instructor-based and peer-oriented attention guidance functionalities in an open source anchored discussion system , 2014, Comput. Educ..

[9]  Roy B. Clariana,et al.  Fostering group norm development and orientation while creating awareness contents for improving net-based collaborative problem solving , 2014, Comput. Hum. Behav..

[10]  Barry Lee Reynolds,et al.  An investigation of the role of article commendation and criticism in Taiwanese university students' heavy BBS usage , 2014, Comput. Educ..

[11]  Terry Ryan,et al.  Enhancing student knowledge acquisition from online learning conversations , 2013, International Journal of Computer-Supported Collaborative Learning.

[12]  R. Sawyer.  The Cambridge Handbook of the Learning Sciences: Introduction , 2014 .

[13]  M. Scardamalia,et al.  Knowledge Building: Theory, Pedagogy, and Technology , 2006 .

[14]  Evren Eryilmaz,et al.  Instructor versus Peer Attention Guidance in Online Learning Conversations , 2015, AIS Trans. Hum. Comput. Interact..

[15]  Victor Kaptelinin,et al.  Group Cognition Computer Support for Building Collaborative Knowledge , 2007 .

[16]  Alyssa Friend Wise,et al.  Attending to others’ posts in asynchronous discussions: Learners’ online “listening” and its relationship to speaking , 2014, Int. J. Comput. Support. Collab. Learn..

[17]  George Siemens,et al.  Penetrating the fog: analytics in learning and education , 2014 .

[18]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[19]  Starr Roxanne Hiltz,et al.  Impacts of college-level courses via Asynchronous Learning Networks: Some Preliminary Results , 2019, Online Learning.

[20]  Veda C. Storey,et al.  Business Intelligence and Analytics: From Big Data to Big Impact , 2012, MIS Q..

[21]  Friedrich W. Hesse,et al.  Using technological functions on a multi-touch table and their affordances to counteract biases and foster collaborative problem solving , 2018, International Journal of Computer-Supported Collaborative Learning.

[22]  野中 郁次郎,et al.  The knowledge-creating company , 2008 .

[23]  Susan Pedersen,et al.  The influence of perceived information overload on student participation and knowledge construction in computer-mediated communication , 2012 .

[24]  Evren Eryilmaz,et al.  How Design Science Research Helps Improve Learning Efficiency in Online Conversations , 2018, Commun. Assoc. Inf. Syst..

[25]  Sebastián Ventura,et al.  Data mining in education , 2013, WIREs Data Mining Knowl. Discov..

[26]  Charlotte N. Gunawardena,et al.  Analysis of a Global Online Debate and the Development of an Interaction Analysis Model for Examining Social Construction of Knowledge in Computer Conferencing , 1997 .

[27]  Robert D. Macredie,et al.  The effects of individual differences and visual instructional aids on disorientation, learning performance and attitudes in a Hypermedia Learning System , 2012, Comput. Hum. Behav..

[28]  Evren Eryilmaz,et al.  Affordances of Recommender Systems for Disorientation in Large Online Conversations , 2019 .

[29]  Christoph Rensing,et al.  Evaluating Recommender Systems for Technology Enhanced Learning: A Quantitative Survey , 2015, IEEE Transactions on Learning Technologies.

[30]  Jim Hewitt,et al.  An investigation of student practices in asynchronous computer conferencing courses , 2010, Comput. Educ..

[31]  Carolyn Penstein Rosé,et al.  Towards Academically Productive Talk Supported by Conversational Agents , 2012, ITS.

[32]  Evren Eryilmaz,et al.  Dynamic Visualization of Quality in Online Conversations , 2018, 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI).