Implementing a Learning Analytics Intervention and Evaluation Framework: what works?

Substantial progress in learning analytics research has been made in recent years to predict which groups of learners are at risk. In this chapter, we argue that the largest challenge for learning analytics research and practice still lies ahead of us: using learning analytics modelling, which types of interventions have a positive impact on learners’ Attitudes, Behaviour and Cognition (ABC). Two embedded case-studies in social science and science are discussed, whereby notions of evidence-based research are illustrated by scenarios (quasi-experimental, A/B-testing, RCT) to evaluate the impact of interventions. Finally, we discuss how a Learning Analytics Intervention and Evaluation Framework (LA-IEF) is currently being implemented at the Open University UK using principles of design-based research and evidence-based research.

[1]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[2]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[3]  R. Slavin.  Evidence-Based Education Policies: Transforming Educational Practice and Research , 2002 .

[4]  Allan Collins,et al.  Design Research: Theoretical and Methodological Issues , 2004 .

[5]  James H. McMillan,et al.  Research in Education: Evidence Based Inquiry , 2005 .

[6]  J. B. Arbaugh,et al.  Is There an Optimal Design for On-Line MBA Courses? , 2005 .

[7]  Andrew J. Martin.  Examining a multidimensional model of student motivation and engagement using a construct validation approach. , 2007, The British journal of educational psychology.

[8]  D. Torgerson,et al.  Designing randomised trials in health, education and the social sciences : an introduction , 2008 .

[9]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[10]  Robert E. Slavin,et al.  Perspectives on Evidence-Based Research in Education—What Works? Issues in Synthesizing Educational Program Evaluations , 2008 .

[11]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[12]  Anne C. Frenzel,et al.  Measuring emotions in students learning and performance: The Achievement Emotions Questionnaire (AE , 2011 .

[13]  Francisco J. García-Peñalvo,et al.  Opening Learning Management Systems to Personal Learning Environments , 2011, J. Univers. Comput. Sci..

[14]  Dirk T. Tempelaar,et al.  How achievement emotions impact students' decisions for online learning, and what precedes those emotions , 2012, Internet High. Educ..

[15]  John T. E. Richardson,et al.  The attainment of White and ethnic minority students in distance education , 2012 .

[16]  John T. E. Richardson,et al.  The role of response biases in the relationship between students’ perceptions of their courses and their approaches to studying in higher education , 2012 .

[17]  Rebecca Ferguson,et al.  Social learning analytics: five approaches , 2012, LAK.

[18]  Dirk T. Tempelaar,et al.  The role of scaffolding and motivation in CSCL , 2012, Comput. Educ..

[19]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[20]  Marie Bienkowski,et al.  Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief , 2012 .

[21]  Hendrik Drachsler,et al.  Translating Learning into Numbers: A Generic Framework for Learning Analytics , 2012, J. Educ. Technol. Soc..

[22]  Bart Rienties,et al.  Integrating ICT in Business Education: Using TPACK to Reflect on Two Course Redesigns , 2012 .

[23]  Sylwia Męcfal.  Recenzja książki. Robert K. yin, Case Study Research. Design and Methods (fourth Edition), thousand Oaks, CA: Sage Publications, 2009 , 2012 .

[24]  Sue Greener,et al.  Interactive learning environments? , 2012, Interact. Learn. Environ..

[25]  Francisco J. García-Peñalvo,et al.  Human-computer interaction in evolutionary visual software analytics , 2013, Comput. Hum. Behav..

[26]  D. Gašević,et al.  “Choose Your Classmates, Your GPA Is at Stake!” , 2013 .

[27]  Simon Buckingham Shum,et al.  Epistemology, pedagogy, assessment and learning analytics , 2013, LAK '13.

[28]  Zdenek Zdráhal,et al.  Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment , 2013, LAK '13.

[29]  Sebastián Ventura,et al.  Data mining in education , 2013, WIREs Data Mining Knowl. Discov..

[30]  Dan Siroker,et al.  A/B Testing: The Most Powerful Way to Turn Clicks Into Customers , 2013 .

[31]  F. Hess,et al.  Breakthrough Leadership in the Digital Age: Using Learning Science to Reboot Schooling , 2013 .

[32]  Dirk T. Tempelaar,et al.  Investigating the relations between motivation, tool use, participation, and performance in an e-learning course using web-videoconferencing , 2013, Comput. Hum. Behav..

[33]  Lorna M. Campbell,et al.  Analytics for Education , 2014 .

[34]  Alyssa Friend Wise,et al.  Designing pedagogical interventions to support student use of learning analytics , 2014, LAK.

[35]  Zdenek Zdráhal,et al.  Developing predictive models for early detection of at-risk students on distance learning modules , 2014, LAK Workshops.

[36]  Agustín C. Caminero,et al.  Analyzing the students' behavior and relevant topics in virtual learning communities , 2014, Comput. Hum. Behav..

[37]  J. B. Arbaugh,et al.  System, scholar or students? Which most influences online MBA course effectiveness? , 2014, J. Comput. Assist. Learn..

[38]  Nicolae Nistor,et al.  Participation in virtual academic communities of practice under the influence of technology acceptance and community factors. A learning analytics application , 2014, Comput. Hum. Behav..

[39]  Doug Clow.  Data wranglers: human interpreters to help close the feedback loop , 2014, LAK '14.

[40]  Nitesh V. Chawla,et al.  Engagement vs performance: using electronic portfolios to predict first semester engineering student retention , 2014, LAK.

[41]  Miguel Ángel Conde González,et al.  Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning , 2014, Comput. Hum. Behav..

[42]  Anastasios A. Economides,et al.  Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence , 2014, J. Educ. Technol. Soc..

[43]  Anastasios A. Economides,et al.  Temporal learning analytics for computer based testing , 2014, LAK.

[44]  Daniel T. Hickey,et al.  Small to big before massive: scaling up participatory learning analytics , 2014, LAK '14.

[45]  Carol Calvert.  Developing a model and applications for probabilities of student success: a case study of predictive analytics , 2014 .

[46]  Dirk T. Tempelaar,et al.  In search for the most informative data for feedback generation: Learning analytics in a data-rich context , 2015, Comput. Hum. Behav..

[47]  Miguel Ángel Conde González,et al.  Learning analytics for educational decision making , 2015, Comput. Hum. Behav..

[48]  Bart Rienties,et al.  "Scaling up" learning design: impact of learning design activities on LMS behavior and performance , 2015, LAK.

[49]  Tom Inkelaar,et al.  Challenging the ‘distance education deficit’ through ‘motivational emails’ , 2015 .

[50]  Bart Rienties,et al.  Why some teachers easily learn to use a new virtual learning environment: a technology acceptance perspective , 2016, Interact. Learn. Environ..

[51]  B. Rienties,et al.  Multi-dimensional transitions of international students to higher education , 2016 .

[52]  Alan J. Dix.  Human-Computer Interaction , 2018, Encyclopedia of Database Systems.

[53]  Dragan Gasevic,et al.  Proceedings of the 8th International Conference on Learning Analytics and Knowledge , 2011, LAK.