How Do We Model Learning at Scale? A Systematic Review of Research on MOOCs

Despite a surge of empirical work on student participation in online learning environments, the causal links between the learning-related factors and processes with the desired learning outcomes remain unexplored. This study presents a systematic literature review of approaches to model learning in Massive Open Online Courses offering an analysis of learning-related constructs used in the prediction and measurement of student engagement and learning outcome. Based on our literature review, we identify current gaps in the research, including a lack of solid frameworks to explain learning in open online setting. Finally, we put forward a novel framework suitable for open online contexts based on a well-established model of student engagement. Our model is intended to guide future work studying the association between contextual factors (i.e., demographic, classroom, and individual needs), student engagement (i.e., academic, behavioral, cognitive, and affective engagement metrics), and learning outcomes (i.e., academic, social, and affective). The proposed model affords further interstudy comparisons as well as comparative studies with more traditional education models.

[1]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[2]  M. Kane The Argument-Based Approach to Validation , 1990 .

[3]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[4]  Philip H. Winne,et al.  Studying as self-regulated learning. , 1998 .

[5]  M. Eraut Non-formal learning and tacit knowledge in professional work. , 2000, The British journal of educational psychology.

[6]  Mirjam James,et al.  Methodological Approaches , 2000, The Monuments of Seti I.

[7]  P. Goodyear Psychological foundations for networked learning , 2001 .

[8]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[9]  George Siemens Connectivism: A Learning Theory for the Digital Age , 2004 .

[10]  Stephen J Ceci,et al.  The rhetoric and reality of gap closing: when the "have-nots" gain but the "haves" gain even more . , 2005, The American psychologist.

[11]  Pamela A. Moss,et al.  Chapter 4: Validity in Educational Assessment , 2006 .

[12]  James J. Appleton,et al.  Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument , 2006 .

[13]  Knud Illeris,et al.  How We Learn: Learning and Non-Learning in School and Beyond , 2007 .

[14]  R. Azevedo,et al.  A Theoretical Review of Winne and Hadwin’s Model of Self-Regulated Learning: New Perspectives and Directions , 2007 .

[15]  Chris Jones Networked Learning - A social practice perspective , 2008 .

[16]  Arthur C. Graesser,et al.  Cohesion Relationships in Tutorial Dialogue as Predictors of Affective States , 2009, AIED.

[17]  Michelene T. H. Chi,et al.  Active-Constructive-Interactive: A Conceptual Framework for Differentiating Learning Activities , 2009, Top. Cogn. Sci..

[18]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[19]  Arthur C. Graesser,et al.  Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments , 2010, Int. J. Hum. Comput. Stud..

[20]  Maite Taboada,et al.  Lexicon-Based Methods for Sentiment Analysis , 2011, CL.

[21]  Maggie Hartnett,et al.  Examining motivation in online distance learning environments: Complex, multifaceted and situation-dependent , 2011 .

[22]  S. Christenson,et al.  Jingle, Jangle, 1 and Conceptual Haziness 2 : Evolution and Future Directions of the Engagement Construct , 2012 .

[23]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[24]  C. Osvaldo Rodriguez,et al.  MOOCs and the AI-Stanford Like Courses: Two Successful and Distinct Course Formats for Massive Open Online Courses. , 2012 .

[25]  Panagiotis Adamopoulos,et al.  What makes a great MOOC? An interdisciplinary analysis of student retention in online courses , 2013, ICIS.

[26]  Mark Warschauer,et al.  Predicting MOOC performance with Week 1 Behavior , 2014, EDM.

[27]  Carolyn Penstein Rosé,et al.  Sentiment Analysis in MOOC Discussion Forums: What does it tell us? , 2014, EDM.

[28]  E. Xing,et al.  Towards an Integration of Text and Graph Clustering Methods as a Lens for Studying Social Interaction in MOOCs , 2014 .

[29]  Ulrike Cress,et al.  Explaining authors’ contribution to pivotal artifacts during mass collaboration in the Wikipedia’s knowledge base , 2014, Int. J. Comput. Support. Collab. Learn..

[30]  Carolyn Penstein Rosé,et al.  Linguistic Reflections of Student Engagement in Massive Open Online Courses , 2014, ICWSM.

[31]  Rebecca Eynon,et al.  Communication patterns in massively open online courses , 2014, Internet High. Educ..

[32]  Lise Getoor,et al.  Learning Latent Engagement Patterns of Students in Online Courses , 2014, AAAI.

[33]  Conrad S. Tucker,et al.  Mining Student-Generated Textual Data In MOOCS And Quantifying Their Effects on Student Performance and Learning Outcomes , 2014 .

[34]  Jane Sinclair,et al.  Learning Technology for Education in Cloud. MOOC and Big Data , 2014, Communications in Computer and Information Science.

[35]  Erik Duval,et al.  Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses , 2014, LAK '14.

[36]  David E. Pritchard,et al.  Correlating skill and improvement in 2 MOOCs with a student's time on tasks , 2014, L@S.

[37]  Andrew D. Ho,et al.  Changing “Course” , 2014 .

[38]  Mark Warschauer,et al.  Social Positioning and Performance in MOOCs , 2014, EDM.

[39]  Linda Corrin,et al.  Visualizing patterns of student engagement and performance in MOOCs , 2014, LAK.

[40]  Lise Getoor,et al.  Uncovering hidden engagement patterns for predicting learner performance in MOOCs , 2014, L@S.

[41]  Sherif Halawa,et al.  Attrition and Achievement Gaps in Online Learning , 2015, L@S.

[42]  Linda Corrin,et al.  Predicting success: how learners' prior knowledge, skills and activities predict MOOC performance , 2015, LAK.

[43]  Joseph Jay Williams,et al.  Beyond Prediction: First Steps Toward Automatic Intervention in MOOC Student Stopout , 2015, EDM 2015.

[44]  Danielle S. McNamara,et al.  Language to Completion: Success in an Educational Data Mining Massive Open Online Class , 2015, EDM.

[45]  Gautam Biswas,et al.  Behavior Prediction in MOOCs using Higher Granularity Temporal Information , 2015, L@S.

[46]  Patrick Jermann,et al.  Identifying Styles and Paths toward Success in MOOCs , 2015, EDM.

[47]  Ming Ming Chiu,et al.  Effects of sequences of socially regulated learning on group performance , 2015, LAK.

[48]  Marek Hatala,et al.  Penetrating the black box of time-on-task estimation , 2015, LAK.

[49]  Deborah L Engle,et al.  Coursera's Introductory Human Physiology Course: Factors That Characterize Successful Completion of a MOOC. , 2015 .

[50]  Dragan Gasevic,et al.  Importance of Theory in Learning Analytics in Formal and Workplace Settings , 2015 .

[51]  Stephanie D. Teasley,et al.  Who You Are or What You Do: Comparing the Predictive Power of Demographics vs. Activity Patterns in Massive Open Online Courses (MOOCs) , 2015, L@S.

[52]  Michael D. Ekstrand,et al.  Teaching Recommender Systems at Large Scale , 2015, ACM Trans. Comput. Hum. Interact..

[53]  A. Wise,et al.  Why Theory Matters More than Ever in the Age of Big Data , 2015, J. Learn. Anal..

[54]  Kalyan Veeramachaneni,et al.  Transfer Learning for Predictive Models in Massive Open Online Courses , 2015, AIED.

[55]  R. Azevedo Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues , 2015 .

[56]  Joseph A. Konstan,et al.  Teaching recommender systems at large scale: evaluation and lessons learned from a hybrid MOOC , 2014, L@S.

[57]  Arthur C. Graesser,et al.  How do you connect?: analysis of social capital accumulation in connectivist MOOCs , 2015, LAK.

[58]  Patrick Jermann,et al.  Conscientious Behaviour, Flexibility and Learning in Massive Open On-Line Courses , 2015 .

[59]  J. Greene,et al.  Predictors of Retention and Achievement in a Massive Open Online Course , 2015 .

[60]  Garry Robins,et al.  Relational event models for social learning in MOOCs , 2015, Soc. Networks.

[61]  Ryan S. Baker,et al.  Content or platform: Why do students complete MOOCs? , 2015 .

[62]  Patrick Jermann,et al.  MOOC Video Interaction Patterns: What Do They Tell Us? , 2015, EC-TEL.

[63]  N. Selwyn,et al.  Massive Open Online Change? Exploring the Discursive Construction of the "MOOC" in Newspapers. , 2015 .

[64]  Katy Jordan,et al.  Massive Open Online Course Completion Rates Revisited: Assessment, Length and Attrition , 2015 .

[65]  J. Vickers,et al.  Relationship between participants’ level of education and engagement in their completion of the Understanding Dementia Massive Open Online Course , 2015, BMC medical education.

[66]  Kenneth R. Koedinger,et al.  Learning is Not a Spectator Sport: Doing is Better than Watching for Learning from a MOOC , 2015, L@S.

[67]  George Siemens,et al.  Preparing for the Digital University: A Review of the History and Current State of Distance, Blended and Online Learning , 2015 .

[68]  Tawanna Dillahunt,et al.  Learn With Friends: The Effects of Student Face-to-Face Collaborations on Massive Open Online Course Activities , 2015, L@S.

[69]  Justine Cassell,et al.  Connecting the Dots: Predicting Student Grade Sequences from Bursty MOOC Interactions over Time , 2015, L@S.

[70]  René F. Kizilcec,et al.  Motivation as a Lens to Understand Online Learners , 2015, ACM Trans. Comput. Hum. Interact..

[71]  Carolyn Penstein Rosé,et al.  Investigating How Student's Cognitive Behavior in MOOC Discussion Forum Affect Learning Gains , 2015, EDM.

[72]  Yoav Bergner,et al.  Methodological Challenges in the Analysis of MOOC Data for Exploring the Relationship between Discussion Forum Views and Learning Outcomes , 2015, EDM.

[73]  Arthur C. Graesser,et al.  Modeling Learners' Social Centrality and Performance through Language and Discourse , 2015, EDM.

[74]  Justin Reich,et al.  Rebooting MOOC Research , 2015, Science.

[75]  Donatella Persico,et al.  Methodological approaches in MOOC research: Retracing the myth of Proteus , 2015, Br. J. Educ. Technol..

[76]  Carolyn Penstein Rosé,et al.  Exploring the Effect of Confusion in Discussion Forums of Massive Open Online Courses , 2015, L@S.

[77]  Dragan Gasevic,et al.  Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success , 2016, Internet High. Educ..

[78]  Rachel B. Baker,et al.  Persistence Patterns in Massive Open Online Courses (MOOCs) , 2015 .

[79]  Dragan Gasevic,et al.  Translating network position into performance: importance of centrality in different network configurations , 2016, LAK.

[80]  L. Czerniewicz,et al.  Learning through engagement: MOOCs as an emergent form of provision , 2016 .

[81]  Dragan Gasevic,et al.  Challenging Assumptions in Learning Analytics , 2016, J. Learn. Anal..

[82]  Justin Reich,et al.  The Civic Mission of MOOCs: Measuring Engagement across Political Differences in Forums , 2016, L@S.

[83]  Andrew J. Saltarelli,et al.  Who Takes MOOCs , 2016 .

[84]  Angela L. Duckworth,et al.  Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning , 2017, Educational psychologist.

[85]  G. Mahr Validation , 2019, Academic Psychiatry.