How should we measure online learning activity

The proliferation of Web-based learning objects makes finding and evaluating resources a considerable hurdle for learners to overcome. While established learning analytics methods provide feedback that can aid learner evaluation of learning resources, the adequacy and reliability of these methods is questioned. Because engagement with online learning is different from other Web activity, it is important to establish pedagogically relevant measures that can aid the development of distinct, automated analysis systems. Content analysis is often used to examine online discussion in educational settings, but these instruments are rarely compared with each other which leads to uncertainty regarding their validity and reliability. In this study, participation in Massive Open Online Course (MOOC) comment forums was evaluated using four different analytical approaches: the Digital Artefacts for Learning Engagement (DiAL-e) framework, Bloom’s Taxonomy, Structure of Observed Learning Outcomes (SOLO) and Community of Inquiry (CoI). Results from this study indicate that different approaches to measuring cognitive activity are closely correlated and are distinct from typical interaction measures. This suggests that computational approaches to pedagogical analysis may provide useful insights into learning processes. Keywords:  CMC; CSCL; content analysis; learning analytics; MOOCs; pedagogical frameworks (Published: 29 July 2016) Citation:  Research in Learning Technology 2016,  24 : 30088 -  http://dx.doi.org/10.3402/rlt.v24.30088

[1]  David E. Millard,et al.  Can You Tell If they're Learning? Using a Pedagogical Framework to Measure Pedagogical Activity , 2015, 2015 IEEE 15th International Conference on Advanced Learning Technologies.

[2]  David W. Carroll,et al.  Patterns of student writing in a critical thinking course: A quantitative analysis , 2007 .

[3]  Simon Atkinson,et al.  “Beyond Content: Developing Transferable Learning Designs with Digital Video Archives.” , 2008 .

[4]  David Kember,et al.  Determining the level of reflective thinking from students' written journals using a coding scheme based on the work of Mezirow , 1999 .

[5]  Jill Willis,et al.  A cognitive processing framework for learning analytics , 2014, LAK.

[6]  M. Tsui,et al.  Applying the Structure of the Observed Learning Outcomes (SOLO) Taxonomy on Student's Learning Outcomes: An empirical study , 2002 .

[7]  Elizabeth Stacey,et al.  A purposive approach to content analysis: Designing analytical frameworks , 2005, Internet High. Educ..

[8]  David Ginat,et al.  SOLO Taxonomy for Assessing Novices' Algorithmic Design , 2015, SIGCSE.

[9]  Millward Brown,et al.  What Is Wearout Anyway , 1998 .

[10]  G. Harry McLaughlin.  Temptations of the flesch , 1974 .

[11]  J. Mezirow.  A Critical Theory of Adult Learning and Education , 1981 .

[12]  Susan C. Herring,et al.  Grammar and Electronic Communication , 2012 .

[13]  Joseph B. Walther,et al.  Selective self-presentation in computer-mediated communication: Hyperpersonal dimensions of technology, language, and cognition , 2007, Comput. Hum. Behav..

[14]  Xin Chen,et al.  Mining Social Media Data for Understanding Students’ Learning Experiences , 2014, IEEE Transactions on Learning Technologies.

[15]  D. Garrison,et al.  Critical thinking, cognitive presence, and computer conferencing in distance education , 2001 .

[16]  Carolyn Penstein Rosé,et al.  Sentiment Analysis in MOOC Discussion Forums: What does it tell us? , 2014, EDM.

[17]  James Daniel Lehman,et al.  The development of a content analysis model for assessing students’ cognitive learning in asynchronous online discussions , 2011 .

[18]  D. Garrison,et al.  Methodological Issues in the Content Analysis of Computer Conference Transcripts , 2007 .

[19]  D. Krathwohl.  A Revision of Bloom's Taxonomy: An Overview , 2002 .

[20]  Marek Hatala,et al.  What is the Source of Social Capital? The Association between Social Network Position and Social Presence in Communities of Inquiry , 2014, EDM.

[21]  Fang Chen,et al.  Cognitive Load Measurement from User's Linguistic Speech Features for Adaptive Interaction Design , 2009, INTERACT.

[22]  B. Bloom.  Taxonomy of educational objectives , 1956 .

[23]  Tammy Schellens,et al.  Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review , 2006, Comput. Educ..

[24]  Dan Cosley,et al.  Feedback for guiding reflection on teamwork practices , 2007, GROUP.

[25]  Karen Littleton,et al.  Epistemology, Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space , 2014, J. Learn. Anal..

[26]  Lise Getoor,et al.  Modeling Learner Engagement in MOOCs using Probabilistic Soft Logic , 2013 .

[27]  Marek Hatala,et al.  Psychological characteristics in cognitive presence of communities of inquiry: A linguistic analysis of online discussions , 2014, Internet High. Educ..

[28]  William Ickes,et al.  Predicting Final Course Performance From Students’ Written Self-Introductions , 2013 .

[29]  Serhat Kocakaya,et al.  A Structural Equation Modeling on Factors of How Experienced Teachers Affect the Students’ Science and Mathematics Achievements , 2014 .

[30]  Michiko Kobayashi.  Using Web 2.0 in Online Learning: What Students Said About VoiceThread , 2013 .

[31]  T. Graepel,et al.  Private traits and attributes are predictable from digital records of human behavior , 2013, Proceedings of the National Academy of Sciences.

[32]  Alexandra Pickett,et al.  Online learner self-regulation: Learning presence viewed through quantitative content- and social network analysis , 2013 .

[33]  S. Nirthanan,et al.  A Comparative Study to Evaluate the Educational Impact of E-Learning Tools on Griffith University Pharmacy Students’ Level of Understanding Using Bloom’s and SOLO Taxonomies , 2014 .

[34]  Lisa M. Weltzer-Ward,et al.  Content analysis coding schemes for online asynchronous discussion , 2011 .

[35]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[36]  Kevin A Hallgren,et al.  Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. , 2012, Tutorials in quantitative methods for psychology.

[37]  Cynthia L. Selfe,et al.  Computer Conferences and Learning: Authority, Resistance, and Internally Persuasive Discourse , 1990 .

[38]  Starr Roxanne Hiltz,et al.  The Impact of a Computerized Conferencing System on Scientific Research Communities. Final Report. Research Report No. 15. , 1981 .

[39]  Peter Shea,et al.  The Community of Inquiry framework meets the SOLO taxonomy: a process‐product model of online learning , 2011 .

[40]  D. Randy Garrison,et al.  The first decade of the community of inquiry framework: A retrospective , 2010, Internet High. Educ..

[41]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[42]  Laurie P. Dringus,et al.  Learning Analytics Considered Harmful. , 2012 .

[43]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[44]  Rebecca Ferguson,et al.  Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs , 2014, EC-TEL.

[45]  Isabell M. Welpe,et al.  I Like, I Cite? Do Facebook Likes Predict the Impact of Scientific Work? , 2015, PloS one.

[46]  Mandy Lupton,et al.  Learning analytics beyond the LMS: the connected learning analytics toolkit , 2015, LAK.