From prediction to impact: evaluation of a learning analytics retention program

Learning analytics research has often been touted as a means to address concerns regarding student retention outcomes. However, few research studies to date, have examined the impact of the implemented intervention strategies designed to address such retention challenges. Moreover, the methodological rigor of some of the existing studies has been challenged. This study evaluates the impact of a pilot retention program. The study contrasts the findings obtained by the use of different methods for analysis of the effect of the intervention. The pilot study was undertaken between 2012 and 2014 resulting in a combined enrolment of 11,160 students. A model to predict attrition was developed, drawing on data from student information system, learning management system interactions, and assessment. The predictive model identified some 1868 students as academically at-risk. Early interventions were implemented involving learning and remediation support. Common statistical methods demonstrated a positive association between the intervention and student retention. However, the effect size was low. The use of more advanced statistical methods, specifically mixed-effect methods explained higher variability in the data (over 99%), yet found the intervention had no effect on the retention outcomes. The study demonstrates that more data about individual differences is required to not only explain retention but to also develop more effective intervention approaches.

[1]  John Patrick Campbell.  Utilizing student data within the course management system to determine undergraduate student academic success: An exploratory study , 2007 .

[2]  E. Kahu.  Framing student engagement in higher education , 2013 .

[3]  Sebastián Ventura,et al.  Predicting students' final performance from participation in on-line discussion forums , 2013, Comput. Educ..

[4]  University Life of Non-Traditional Students: Engagement Styles and Impact on Attrition , 2011 .

[5]  Simon Brooman,et al.  Measuring the beginning: a quantitative study of the transition to higher education , 2014 .

[6]  Kate Miller,et al.  Better Than Expected: Using Learning Analytics to Promote Student Success in Gateway Science , 2014 .

[7]  M. Nichols,et al.  Student perceptions of support services and the influence of targeted interventions on retention in distance education , 2010 .

[8]  Simon A. Fraser.  Does Time-on-task Estimation Matter? Implications for the Validity of Learning Analytics Findings , 2016 .

[9]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[10]  Shinichi Nakagawa,et al.  A general and simple method for obtaining R2 from generalized linear mixed‐effects models , 2013 .

[11]  S. Kift,et al.  Transition pedagogy: A third generation approach to FYE - A case study of policy and practice for the higher education sector , 2010 .

[12]  Ryan S. Baker,et al.  Educational Data Mining and Learning Analytics , 2014 .

[13]  Sebastián Ventura,et al.  Educational Data Mining: A Review of the State of the Art , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[14]  Elena N. Zaitseva,et al.  Connecting the dots: using concept maps for interpreting student satisfaction , 2013 .

[15]  Eitel J. M. Lauría,et al.  Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative , 2014, J. Learn. Anal..

[16]  Dragan Gasevic,et al.  Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success , 2016, Internet High. Educ..

[17]  Steven Lonn,et al.  A Learning Management System-Based Early Warning System for Academic Advising in Undergraduate Engineering , 2014 .

[18]  Matthew D. Pistilli,et al.  Course signals at Purdue: using learning analytics to increase student success , 2012, LAK.

[19]  References , 1971 .

[20]  D. Bates,et al.  Linear Mixed-Effects Models using 'Eigen' and S4 , 2015 .

[21]  G. Ozakinci,et al.  Understanding undergraduate student perceptions of mental health, mental well-being and help-seeking behaviour , 2016 .

[22]  Dragan Gasevic,et al.  Generating actionable predictive models of academic performance , 2016, LAK.

[23]  Rebecca Ferguson,et al.  LAK failathon , 2016, LAK.

[24]  L. Morris,et al.  Tracking student behavior, persistence, and achievement in online courses , 2005, Internet High. Educ..

[25]  Sebastián Ventura,et al.  Web usage mining for predicting final marks of students that use Moodle courses , 2013, Comput. Appl. Eng. Educ..

[26]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[27]  Kimberly E. Arnold.  Signals: Applying Academic Analytics. , 2010 .

[28]  Andrew F. Hayes,et al.  A Primer on Multilevel Modeling , 2006 .

[29]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[30]  Mike Sharkey,et al.  Course correction: using analytics to predict course success , 2012, LAK '12.

[31]  Marek Hatala,et al.  Penetrating the black box of time-on-task estimation , 2015, LAK.

[32]  Zdenek Zdráhal,et al.  Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment , 2013, LAK '13.