Improving Learning Analytics - Combining Observational and Self-Report Data on Student Learning

Introduction Using learning analytics as a tool to improve student learning has caught the imagination and research effort of much of the higher education sector (Siemens, 2013). Amongst a number of applications, it notably has been used to improve student success (Arnold, Hall, Street, Lafayette, & Pistilli, 2012; Martin et al., 2013), to better understand the nature of social learning amongst university students (Buckingham Shum & Ferguson, 2012), to improve approaches to learning design (Mor, Ferguson, & Wasson, 2015), and to guide university education strategy (Rientes et al., 2016). Accompanying this growing use of learning analytics, there is serious debate about the extent to which they are useful as a tool for improving student learning (Lodge & Lewis, 2012; Lundie, 2014). One debate is about the objectivity of learning analytics; some argue that learning analytics are an objective measure of student activity, but others suggest that without understanding student intent behind the analytics, we have a poor context in which to interpret what the numbers mean (Boyd & Crawford, 2012). Another debate is that learning analytics tell us what students are doing when they learn in an online environment. Doubters argue that they only tell us what buttons they are clicking (Scheffel, Drachsler, Stoyanov, & Specht, 2014). A further debate surrounds the value of very large data sets. Some argue that the more analytics you have about student learning experiences the better, while others argue that a careful selection of analytics must be made in relation to the population sample, otherwise the additional metrics might just create noise in interpreting their meaning. As some studies suggest, indiscriminate approaches to the use of large datasets could lead to unintended consequences in learning interventions (Boyd & Crawford, 2012; Greller & Drachsler, 2012). To remedy some of the perceived shortfalls of learning analytics, some authors argue that the learning analytics should occupy a middle space, somewhere between learning theory and computational measurement, to improve the potential of learning analytics to really address concerns of the quality of student learning (Suthers & Vebert, 2013). To achieve this, they recommend that additional analytic techqniues accompany learning analytic procedures from such fields as epistomology and education studies. To investigate methodological approaches to address some of the perceived shortfalls of learning analytics, this study investigates the first year experience of undergraduate engineering students in a blended course in two stages. In the first stage, it records their learning events in the online environment and analyses and interprets them in the context of their learning outcomes (Pardo, Han, & Ellis, 2016). While illuminating, this analysis alone could be left open to some of the criticisms described above. In the second stage, methodological approaches from Student Approaches to Learning (Pintrich, 2004) are used and the students' response to closed ended questionnaires (Biggs, Kember, & Leung, 2001) about their experience of learning is investigated. The outcomes of this analysis, when complemented by stage 1, both elucidates why some students are relatively more successful than others in the course and provides evidence which suggests why this might be the case. The purpose of this study is to contribute to the international debate on the value of learning analytics for the quality of the student learning experience and how combined methodological approaches using observational and self-report evidence can improve our understanding of qualitative variation in student learning. By drawing on both types of data from the same experience of learning, this study is designed to see to what extent a combined use of the observational and self-report data improves our ability to use learning analytics to understand why some students are more successful than others. …