The effect of providing learning analytics on student behaviour and performance in programming: a randomised controlled experiment

We use a randomised experiment to study the effect of offering half of 556 freshman students a learning analytics dashboard and a weekly email with a link to their dashboard, on student behaviour in the online environment and final exam performance. The dashboard shows their online progress in the learning management systems, their predicted chance of passing, their predicted grade and their online intermediate performance compared with the total cohort. The email with dashboard access, as well as dashboard use, has positive effects on student behaviour in the online environment, but no effects are found on student performance in the final exam of the programming course. However, we do find differential effects by specialisation and student characteristics.

[1]  Stephanie D. Teasley.  Student Facing Dashboards: One Size Fits All? , 2017, Technol. Knowl. Learn..

[2]  Katheryn R. Christy,et al.  Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women's math performance , 2014, Comput. Educ..

[3]  Maren Scheffel,et al.  Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational Practice , 2017, EC-TEL.

[4]  Bernie Dodge,et al.  Improving undergraduate student achievement in large blended courses through data-driven interventions , 2015, LAK.

[5]  R. Schwartz,et al.  Pathways to prosperity: Meeting the challenge of preparing young Americans for the 21st century , 2011 .

[6]  A. V. Nageswararao Rao.  Effectiveness of Blended Learning. , 2006 .

[7]  Jesse Fox,et al.  Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance , 2015, Comput. Educ..

[8]  Nagore Iriberri,et al.  The Importance of Relative Performance Feedback Information: Evidence from a Natural Experiment Using High School Students , 2009 .

[9]  Erik Duval,et al.  Learning Analytics Dashboard Applications , 2013 .

[10]  FoxJesse,et al.  Assessing the effects of gamification in the classroom , 2015 .

[11]  Katrien Verbert,et al.  Data Driven Approaches in Digital Education , 2017, Lecture Notes in Computer Science.

[12]  Philip Oreopoulos,et al.  Thrivers and Divers: Using Non-Academic Measures to Predict College Success and Failure , 2016 .

[13]  Hoisoo Kim,et al.  Effects of learning Strategy, Metacognition and Steps of Multimedia Application on Achievement and Learning Attitude in a Didactic Classroom Teaching , 1999 .

[14]  Pat Langley,et al.  Induction of One-Level Decision Trees , 1992, ML.

[15]  Il-Hyun Jo,et al.  Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement , 2015, Asia Pacific Education Review.

[16]  Edward R. Tufte,et al.  Envisioning Information , 1990 .

[17]  Chia-Lun Lo,et al.  Developing early warning systems to predict students' online learning performance , 2014, Comput. Hum. Behav..

[18]  Eitel J. M. Lauría,et al.  Open academic analytics initiative: initial research findings , 2013, LAK '13.

[19]  Imran Rasul,et al.  ignorance ? : a natural experiment on the effect of feedback on students ' performance , 2015 .

[20]  Shui-fong Lam,et al.  The effects of competition on achievement motivation in Chinese classrooms. , 2004, The British journal of educational psychology.

[21]  R. Murnane,et al.  Methods Matter: Improving Causal Inference in Educational and Social Science Research , 2010 .

[22]  Dirk T. Tempelaar,et al.  In search for the most informative data for feedback generation: Learning analytics in a data-rich context , 2015, Comput. Hum. Behav..

[23]  Erik Duval,et al.  Dataset-Driven Research to Support Learning and Knowledge Analytics , 2012, J. Educ. Technol. Soc..

[24]  Michael J. Doughty,et al.  ATLAS OF OPHTHALMIC SURGERY , 1991 .

[25]  Philip Oreopoulos,et al.  Student Coaching: How Far Can Technology Go? , 2016, The Journal of Human Resources.

[26]  Katrien Verbert,et al.  Trends and issues in student-facing learning analytics reporting systems research , 2017, LAK.

[27]  A. Winsler,et al.  Self-Regulation and Ability Predictors of Academic Success During College: A Predictive Validity Study , 2008 .

[28]  Erik Duval,et al.  Learning dashboards: an overview and future research opportunities , 2013, Personal and Ubiquitous Computing.

[29]  Eva Durall,et al.  Learning Analytics as a Metacognitive Tool , 2014, CSEDU.