Methodological Challenges in the Analysis of MOOC Data for Exploring the Relationship between Discussion Forum Views and Learning Outcomes

Determining how learners use MOOCs effectively is critical to providing feedback to instructors, schools, and policy-makers on this highly scalable technology. However, drawing inferences about student learning outcomes in MOOCs has proven to be quite difficult due to large amounts of missing data (of various kinds) and to the diverse population of MOOC participants. Thus significant methodological challenges must be addressed before seemingly straightforward substantive questions can be answered. The present study considers modeling final exam performance outcomes on early-stage ability estimates, discussion forum viewing frequency, and overall assessment-oriented engagement (AOE, seen as a proxy measure of motivation). These variables require careful operationalization, analysis of which is the principle contribution of this work. This study demonstrates that the effect sizes of discussion forum viewing activities on final exam outcomes are quite sensitive to these choices. Author

[1]  Gerd Kortemeyer,et al.  Correlations between Student Discussion Behavior, Attitudes, and Learning. , 2007 .

[2]  Björn Hartmann,et al.  Chatrooms in MOOCs: all talk and no action , 2014, L@S.

[3]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[4]  Chris Piech,et al.  Deconstructing disengagement: analyzing learner subpopulations in massive open online courses , 2013, LAK '13.

[5]  Martin A. Andresen.  Asynchronous Discussion Forums: Success Factors, Outcomes, Assessments, and Limitations , 2009, J. Educ. Technol. Soc..

[6]  Amy Soller,et al.  Supporting Social Interaction in an Intelligent Collaborative Learning System , 2001 .

[7]  Anirban Dasgupta,et al.  Superposter behavior in MOOC forums , 2014, L@S.

[8]  Errol Yudko,et al.  "Hits" (not "Discussion Posts") predict student success in online courses: A double cross-validation study , 2008, Comput. Educ..

[9]  Robin H. Kay,et al.  Developing a comprehensive metric for assessing discussion board effectiveness , 2006, Br. J. Educ. Technol..

[10]  Marek Hatala,et al.  Penetrating the black box of time-on-task estimation , 2015, LAK.

[11]  Michael A. Conte.  Participation and performance , 1986 .

[12]  D. Eignor.  The standards for educational and psychological testing. , 2013 .

[13]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[14]  Justin Reich,et al.  HarvardX and MITx: The First Year of Open Online Courses, Fall 2012-Summer 2013 , 2014 .

[15]  Yoav Bergner,et al.  Who does what in a massive open online course? , 2014, Commun. ACM.

[16]  Christine E. DeMars,et al.  Low Examinee Effort in Low-Stakes Assessment: Problems and Potential Solutions , 2005 .

[17]  Yoav Bergner,et al.  Estimation of ability from homework items when there are missing and/or multiple attempts , 2015, LAK.

[18]  Andrew D. Ho,et al.  Changing “Course” , 2014 .

[19]  Paul D. McNicholas,et al.  Model-Based Clustering , 2016, Journal of Classification.

[20]  M. J. W. Thomas,et al.  Learning within incoherent structures: the space of online discussion forums , 2002, J. Comput. Assist. Learn..

[21]  Rebecca Eynon,et al.  Communication patterns in massively open online courses , 2014, Internet High. Educ..

[22]  Doug Clow,et al.  MOOCs and the funnel of participation , 2013, LAK '13.

[23]  Liyan Song,et al.  Understanding Students' Online Interaction: Analysis of Discussion Board Postings , 2011 .

[24]  Isaac L. Chuang,et al.  Participation And performance In 8.02x Electricity And Magnetism: The First Physics MOOC From MITx , 2013, 1310.3173.

[25]  Laurie P. Dringus,et al.  Using data mining as a strategy for assessing asynchronous discussion forums , 2005, Comput. Educ..

[26]  Derek Rowntree,et al.  Teaching and learning online: a correspondence education for the 21st century? , 1995, Br. J. Educ. Technol..

[27]  J. Eccles,et al.  Motivational beliefs, values, and goals. , 2002, Annual review of psychology.

[28]  A W Tony Bates,et al.  Technology, e-learning, and distance education , 1995 .

[29]  Hanna Eklöf,et al.  Development and Validation of Scores From an Instrument Measuring Student Test-Taking Motivation , 2006 .