Replicating MOOC predictive models at scale

We present a case study in predictive model replication for student dropout in Massive Open Online Courses (MOOCs) using a large and diverse dataset (133 sessions of 28 unique courses offered by two institutions). This experiment was run on the MOOC Replication Framework (MORF), which makes it feasible to fully replicate complex machine learned models, from raw data to model evaluation. We provide an overview of the MORF platform architecture and functionality, and demonstrate its use through a case study. In this replication of [41], we contextualize and evaluate the results of the previous work using statistical tests and a more effective model evaluation scheme. We find that only some of the original findings replicate across this larger and more diverse sample of MOOCs, with others replicating significantly in the opposite direction. Our analysis also reveals results which are highly relevant to the prediction task which were not reported in the original experiment. This work demonstrates the importance of replication of predictive modeling research in MOOCs using large and diverse datasets, illuminates the challenges of doing so, and describes our freely available, open-source software framework to overcome barriers to replication.

[1]  Ryan Shaun Joazeiro de Baker,et al.  MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[2]  A. Gelman,et al.  The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗ , 2019 .

[3]  Harald C. Gall,et al.  Using Docker Containers to Improve Reproducibility in Software and Web Engineering Research , 2016, ICWE.

[4]  Danielle S. McNamara,et al.  Combining click-stream data with NLP tools to better understand MOOC completion , 2016, LAK.

[5]  Christopher Brooks,et al.  Dropout Model Evaluation in MOOCs , 2018, AAAI.

[6]  Jacob Whitehill,et al.  Delving Deeper into MOOC Student Dropout Prediction , 2017, ArXiv.

[7]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[8]  Ryan Shaun Joazeiro de Baker,et al.  MORF: A Framework for MOOC Predictive Modeling and Replication At Scale , 2018, ArXiv.

[9]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[10]  Patrick Jermann,et al.  Capturing "attrition intensifying" structural traits from didactic interaction sequences of MOOC learners , 2014, EMNLP 2014.

[11]  Zachary A. Pardos,et al.  MOOCdb: Developing Data Standards for MOOC Data Science , 2013 .

[12]  Vincent Tinto,et al.  Research and Practice of Student Retention: What Next? , 2006 .

[13]  Josh Gardner,et al.  Student success prediction in MOOCs , 2017, User Modeling and User-Adapted Interaction.

[14]  George Siemens,et al.  Replicating 21 findings on student success in online learning , 2017 .

[15]  George Siemens,et al.  Studying MOOC completion at scale using the MOOC replication framework , 2018, EDM.

[16]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[17]  Girish Balakrishnan,et al.  Predicting Student Retention in Massive Open Online Courses using Hidden Markov Models , 2013 .

[18]  Stephanie D. Teasley,et al.  A time series interaction analysis method for building predictive models of learners using log data , 2015, LAK.

[19]  Christopher Brooks,et al.  Diverse Big Data and Randomized Field Experiments in MOOCs , 2017 .

[20]  George Sugai,et al.  Using Sixth-Grade School Records to Predict School Violence, Chronic Discipline Problems, and High School Outcomes , 1999 .

[21]  Christopher Brooks,et al.  Evaluating Predictive Models of Student Success: Closing the Methodological Gap , 2018, J. Learn. Anal..

[22]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[23]  Justin Reich,et al.  Privacy, anonymity, and big data in the social sciences , 2014, Commun. ACM.

[24]  Dit-Yan Yeung,et al.  Temporal Models for Predicting Student Dropout in Massive Open Online Courses , 2015, 2015 IEEE International Conference on Data Mining Workshop (ICDMW).

[25]  Kalyan Veeramachaneni,et al.  Towards Feature Engineering at Scale for Data from Massive Open Online Courses , 2014, ArXiv.

[26]  Sherif Halawa,et al.  Attrition and Achievement Gaps in Online Learning , 2015, L@S.

[27]  Rachel B. Baker,et al.  Persistence Patterns in Massive Open Online Courses (MOOCs) , 2015 .

[28]  Kalyan Veeramachaneni,et al.  Transfer Learning for Predictive Models in Massive Open Online Courses , 2015, AIED.

[29]  Carolyn Penstein Rosé,et al.  “ Turn on , Tune in , Drop out ” : Anticipating student dropouts in Massive Open Online Courses , 2013 .

[30]  Jeffrey R. Spies,et al.  The Replication Recipe: What Makes for a Convincing Replication? , 2014 .

[31]  Carl Boettiger,et al.  An introduction to Docker for reproducible research , 2014, OPSR.

[32]  Ryan Shaun Joazeiro de Baker,et al.  Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction , 2005, Graphics Interface.

[33]  Jacob Whitehill,et al.  MOOC Dropout Prediction: How to Measure Accuracy? , 2017, L@S.

[34]  Xin Chen,et al.  Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization , 2016, Comput. Hum. Behav..

[35]  Harald C. Gall,et al.  Using Docker Containers to Improve Reproducibility in Software Engineering Research , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[36]  D. Donoho.  50 Years of Data Science , 2017 .

[37]  Matthew C. Makel,et al.  Facts Are More Important Than Novelty , 2014 .

[38]  Jose Miguel Puerta,et al.  Learning Bayesian networks by hill climbing: efficient methods based on progressive restriction of the neighborhood , 2010, Data Mining and Knowledge Discovery.

[39]  Victoria Stodden,et al.  Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research , 2014 .

[40]  Dirk Merkel,et al.  Docker: lightweight Linux containers for consistent development and deployment , 2014 .

[41]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[42]  S. Stouffer.  Adjustment during army life , 1977 .

[43]  Caroline Jay,et al.  The Blind Men and the Elephant: Towards an Empirical Evaluation Framework for Software Sustainability , 2014 .

[44]  Christian Collberg,et al.  Measuring Reproducibility in Computer Systems Research , 2014 .