How learning analytics can early predict under-achieving students in a blended medical education course

Abstract Aim: Learning analytics (LA) is an emerging discipline that aims at analyzing students’ online data in order to improve the learning process and optimize learning environments. It has yet un-explored potential in the field of medical education, which can be particularly helpful in the early prediction and identification of under-achieving students. The aim of this study was to identify quantitative markers collected from students’ online activities that may correlate with students’ final performance and to investigate the possibility of predicting the potential risk of a student failing or dropping out of a course. Methods: This study included 133 students enrolled in a blended medical course where they were free to use the learning management system at their will. We extracted their online activity data using database queries and Moodle plugins. Data included logins, views, forums, time, formative assessment, and communications at different points of time. Five engagement indicators were also calculated which would reflect self-regulation and engagement. Students who scored below 5% over the passing mark were considered to be potentially at risk of under-achieving. Results: At the end of the course, we were able to predict the final grade with 63.5% accuracy, and identify 53.9% of at-risk students. Using a binary logistic model improved prediction to 80.8%. Using data recorded until the mid-course, prediction accuracy was 42.3%. The most important predictors were factors reflecting engagement of the students and the consistency of using the online resources. Conclusions: The analysis of students’ online activities in a blended medical education course by means of LA techniques can help early predict underachieving students, and can be used as an early warning sign for timely intervention.

[1]  Anastasios A. Economides,et al.  Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence , 2014, J. Educ. Technol. Soc..

[2]  George Siemens,et al.  Let’s not forget: Learning analytics are about learning , 2015 .

[3]  Nabil Zary,et al.  Big Data in Medical Informatics: Improving Education Through Visual Analytics , 2014, MIE.

[4]  Shane Dawson,et al.  Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan , 2012, J. Educ. Technol. Soc..

[5]  Matthew D. Pistilli,et al.  In practice: Purdue Signals: Mining real‐time academic data to enhance student success , 2010 .

[6]  A. Wise,et al.  Why Theory Matters More than Ever in the Age of Big Data , 2015, J. Learn. Anal..

[7]  Mithat Gönen,et al.  Receiver Operating Characteristic (ROC) Curves , 2006 .

[8]  Shane Dawson,et al.  Informing Pedagogical Action , 2013 .

[9]  Ewout W Steyerberg,et al.  Predicting performance: relative importance of students’ background and past performance , 2015, Medical education.

[10]  I. Doherty,et al.  Contemporary and future eLearning trends in medical education , 2015, Medical teacher.

[11]  P. Panzarasa,et al.  Temporal patterns and dynamics of e-learning usage in medical education , 2016 .

[12]  George Siemens,et al.  Learning Analytics , 2013 .

[13]  K. Masters,et al.  AMEE Guide 32: e-Learning in medical education Part 1: Learning, teaching and assessment , 2008, Medical teacher.

[14]  M. Pusic,et al.  Developing the role of big data and analytics in health professional education , 2014, Medical teacher.

[15]  Birgitta Wallstedt,et al.  Factors associated with dropout in medical education: a literature review , 2011, Medical education.

[16]  Catherine L. Finnegan,et al.  Differences by Course Discipline on Student Behavior, Persistence, and Achievement in Online Courses of Undergraduate General Education , 2008 .

[17]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[18]  Jared Dean,et al.  Big Data, Data Mining, and Machine Learning: Value Creation for Business Leaders and Practitioners , 2014 .

[19]  Murtaza Haider,et al.  Beyond the hype: Big data concepts, methods, and analytics , 2015, Int. J. Inf. Manag..

[20]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[21]  Weirong Yan,et al.  The Effectiveness of Blended Learning in Health Professions: Systematic Review and Meta-Analysis , 2016, Journal of medical Internet research.

[22]  Zdenek Zdráhal,et al.  Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment , 2013, LAK '13.

[23]  Zdenek Zdráhal,et al.  Developing predictive models for early detection of at-risk students on distance learning modules , 2014, LAK Workshops.

[24]  C. Y. Peng,et al.  An Introduction to Logistic Regression Analysis and Reporting , 2002 .

[25]  Ben Shneiderman,et al.  Analyzing Social Media Networks with NodeXL: Insights from a Connected World , 2010 .

[26]  Miguel Ángel Conde González,et al.  Learning analytics for educational decision making , 2015, Comput. Hum. Behav..

[27]  Bart Rienties,et al.  Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK , 2016 .

[28]  Miguel Ángel Conde González,et al.  Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning , 2014, Comput. Hum. Behav..

[29]  Viv Bewick,et al.  Statistics review 14: Logistic regression , 2005, Critical care.

[30]  Dragan Gasevic,et al.  Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success , 2016, Internet High. Educ..

[31]  Alexandra Pickett,et al.  Online learner self-regulation: Learning presence viewed through quantitative content- and social network analysis , 2013 .

[32]  Dirk T. Tempelaar,et al.  In search for the most informative data for feedback generation: Learning analytics in a data-rich context , 2015, Comput. Hum. Behav..

[33]  Matthew D. Pistilli,et al.  Course signals at Purdue: using learning analytics to increase student success , 2012, LAK.

[34]  David C. Hoaglin,et al.  Exploratory Data Analysis , 2005 .

[35]  Hongwei Yang,et al.  The Case for Being Automatic: Introducing the Automatic Linear Modeling (LINEAR) Procedure in SPSS Statistics , 2013 .

[36]  Maria Meehan,et al.  Developing Accurate Early Warning Systems Via Data Analytics , 2016 .

[37]  P. Butler,et al.  Learner enhanced technology , 2016 .

[38]  Francisco J. García-Peñalvo,et al.  Discovering usage behaviors and engagement in an Educational Virtual World , 2015, Comput. Hum. Behav..

[39]  Ara Tekian,et al.  Overcome the 60% passing score and improve the quality of assessment , 2015, GMS Zeitschrift fur medizinische Ausbildung.

[40]  Jafar Habibi,et al.  Using Educational Data Mining Methods to Study the Impact of Virtual Classroom in E-Learning , 2010, EDM.

[41]  D. Christopher Brooks,et al.  The Current Ecosystem of Learning Management Systems in Higher Education: Student, Faculty, and IT Perspectives. , 2014 .

[42]  Christopher Cheong,et al.  Designing Persuasive Systems to Influence Learning: Modelling the Impact of Study Habits on Academic Performance , 2015, PACIS.

[43]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[44]  Malcolm E. Brown,et al.  Learning Analytics : The Coming Third Wave , 2011 .

[45]  J. Norcini,et al.  Setting standards on educational tests , 2003, Medical education.

[46]  Errol Yudko,et al.  "Hits" (not "Discussion Posts") predict student success in online courses: A double cross-validation study , 2008, Comput. Educ..