Scholarly insight 2016: a Data wrangler perspective

We are pleased to offer you our first Scholarly insight 2016: a Data wrangler perspective. The OU is going through several fundamental changes, whereby strategic, pedagogical informed research and insight what drives student learning and academic performance is essential. Making sense of Big Data in particular can be a challenge, especially when data is stored at different data warehouses and require advanced statistical skills to interpret complex patterns of data. In 2012 the Open University UK (OU) instigated a Data Wrangling initiative, which provided every Faculty with a dedicated academic with expertise in data analysis and whose task is to provide strategic, pedagogical, and sense-making advice to staff and senior management. Given substantial changes within the OU over the last 18 months (e.g., new Faculty structure, real-time dashboards, increased reliance on analytics), an extensive discussion with various stakeholders within the Faculties was initiated to make sure that data wranglers provide effective pedagogical insight based upon best practice and evidence-based analyses and research (see new Data wrangler structure). Demand for actionable insights to help support OU staff and senior management in particular with module and qualification design is currently strong (Miller & Mork, 2013), especially a desire for evidence of impact of “what works” (Ferguson, Brasher, et al., 2016). Learning analytics are now increasingly taken into consideration when designing, writing and revising modules, and in the evaluation of specific teaching approaches and technologies (Rienties, Boroowa, et al., 2016). A range of data interrogation and visualization tools developed by the OU supports this (Calvert, 2014; Toetenel & Rienties, 2016b). With the new ways of working with Data Wrangling, first we have provided our basic statistical analyses in form of our Key Metrics report. Second, from January 2017 onwards we will focus again on dealing with bespoke requests from Faculties, and where possible share the insights across all Schools and Faculties. Third, this Scholarly insight has a different purpose to previous Data wrangler work, namely we aim to provide state-of-the-art and forward looking insights into what drives our students and staff in terms of learning and learning success. Based upon consultation with the Faculties, seven key cross-Faculty themes were identified that influence our students’ learning experiences, academic performance, and retention. The first five chapters focus on how the OU designs modules, formative and summative assessments and feedback, helps students from informal to formal learning, and how these learning designs influence student satisfaction. All five chapters indicate that the way we design our modules fundamentally influences student satisfaction, and perhaps more importantly academic retention. Clear guidelines and good-reads are provided for how module teams, ALs, and others can improve our focus on Students First. In Chapter 6-7, we specifically address how individual student demographics (e.g., age, ethnicity, prior education) and accessibility in particular influence the students’ learning journeys, with concrete suggestions how to support our diverse groups of students. Note that each chapter can be read independently and in any particular order. We are looking forward to your feedback.

[1]  John T. E. Richardson,et al.  Perceptions of academic quality and approaches to studying among disabled and nondisabled students in distance education , 2010 .

[2]  Benoit Guerin,et al.  Learning Gain in Higher Education , 2021, International Perspectives on Higher Education Research.

[3]  R. Berenson,et al.  Emotional Intelligence as a Predictor for Success in Online Learning , 2008 .

[4]  John Garry,et al.  Are Moderators Moderate?: Testing the ‘Anchoring and Adjustment’ Hypothesis in the Context of Marking Politics Exams 1 , 2005 .

[5]  Anne Jelfs,et al.  Ten years of open practice: a reflection on the impact of OpenLearn , 2016 .

[6]  J. Arbaugh,et al.  A Structural Equation Model of Predictors for Effective Online Learning , 2005 .

[7]  D. Nicol,et al.  Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice , 2006 .

[8]  John T. E. Richardson,et al.  The attainment and experiences of disabled students in distance education , 2009 .

[9]  Eileen Scanlon,et al.  Designing for Educational Technology to Enhance the Experience of Learners in Distance Education: How Open Educational Resources, Learning Design and Moocs Are Influencing Learning , 2015 .

[10]  Bart Rienties,et al.  Unravelling the dynamics of instructional practice: a longitudinal study on learning design and VLE activities , 2017, LAK.

[11]  Tim Coughlan,et al.  Assessing OER impact across organisations and learners: experiences from the Bridge to Success project , 2013 .

[12]  Dirk T. Tempelaar,et al.  Adding dispositions to create pedagogy-based Learning Analytics , 2017 .

[13]  D. Bligh Learning to teach in higher education , 1993 .

[14]  Uta Boehm,et al.  Designing For Learning In An Open World , 2016 .

[15]  C. Rust The Unscholarly Use of Numbers in Our Assessment Practices: What Will Make Us Change? , 2011 .

[16]  Alex Buckley The UK Engagement Survey 2014: The Second Pilot Year , 2014 .

[17]  D. Boud,et al.  Aligning assessment with long‐term learning , 2006 .

[18]  Susanne Narciss Designing and evaluating tutoring Feedback Strategies for Digital Learning , 2013 .

[19]  Mike Phillips,et al.  Engaging large groups of individual learners through an online environment – an emerging i-Learner generation? , 2013 .

[20]  Bart Rienties,et al.  Implementing a Learning Analytics Intervention and Evaluation Framework: what works? , 2017 .

[21]  Rebecca Ferguson,et al.  Moving Through MOOCS: Pedagogy, Learning Design and Patterns of Engagement , 2015, EC-TEL.

[22]  Bart Rienties,et al.  Understanding academics’ resistance towards (online) student evaluation , 2014 .

[23]  Martin Weller,et al.  Final Project Report of the OULDI- JISC Project: Challenge and Change in Curriculum Design Process, Communities, Visualisation and Practice , 2012 .

[24]  Rebecca Ferguson,et al.  Innovating Pedagogy 2016: Open University Innovation Report 5 , 2016 .

[25]  P. Ramsden A performance indicator of teaching quality in higher education: The Course Experience Questionnaire , 1991 .

[26]  Rebecca Ferguson,et al.  Social learning analytics: five approaches , 2012, LAK.

[27]  Alan Roulstone,et al.  Support for Higher Education Students with Specific Learning Difficulties: Report to HEFCE by York Consulting and University of Leeds , 2015 .

[28]  Richard Hays,et al.  Evaluating teaching and learning , 2018 .

[29]  Denise Whitelock,et al.  The promise and potential of e-assessment for learning , 2015 .

[30]  L. English Global Perspectives on Recognising Non-formal and Informal Learning:Why Recognition Matters , 2015 .

[31]  J. Stephenson Visible learning and the science of how we learn , 2014 .

[32]  David Sloan,et al.  Contextual web accessibility - maximizing the benefit of accessibility guidelines , 2006, W4A '06.

[33]  J. B. Arbaugh,et al.  System, scholar or students? Which most influences online MBA course effectiveness? , 2014, J. Comput. Assist. Learn..

[34]  Katy Jordan,et al.  Massive Open Online Course Completion Rates Revisited: Assessment, Length and Attrition , 2015 .

[35]  Elizabeth A Gazza,et al.  Facilitating student retention in online graduate nursing education programs: a review of the literature. , 2014, Nurse education today.

[36]  Denise Whitelock,et al.  #Design4Learning: Designing for the Future of Higher Education , 2016 .

[37]  Margaret Heritage,et al.  Formative Assessment: What Do Teachers Need to Know and Do? , 2007 .

[38]  David Sloan,et al.  A challenge to web accessibility metrics and guidelines: putting people and processes first , 2012, W4A.

[39]  Dirk T. Tempelaar,et al.  In search for the most informative data for feedback generation: Learning analytics in a data-rich context , 2015, Comput. Hum. Behav..

[40]  D. Rowntree Assessing Students: How Shall We Know Them? , 1977 .

[41]  Denise Whitelock,et al.  Designing and testing visual representations of draft essays for higher education students , 2014 .

[42]  B. Rienties,et al.  Online learning experiences of new versus continuing learners: a large-scale replication study , 2017 .

[43]  Bart Rienties,et al.  Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK , 2016 .

[44]  Rebecca Ferguson,et al.  What can analytics contribute to accessibility in e-learning systems and to disabled students' learning? , 2016, LAK.

[45]  Miguel Ángel Conde González,et al.  Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning , 2014, Comput. Hum. Behav..

[46]  Lambert Schuwirth,et al.  Optimising new modes of assessment: in search of qualities and standards , 2004 .

[47]  Martin Hlosta,et al.  OU Analyse: analysing at-risk students at The Open University , 2015 .

[48]  Donatella Persico,et al.  Self-Regulated Learning , 2011 .

[49]  Sean B. Eom,et al.  The Determinants of Students' Perceived Learning Outcomes and Satisfaction in University Online Education: An Empirical Investigation* , 2006 .

[50]  Shaobo Huang,et al.  Predicting student academic performance in an engineering dynamics course: A comparison of four types of predictive mathematical models , 2013, Comput. Educ..

[51]  J. Beishuizen,et al.  Student learning experience as indicator of teaching quality , 2012 .

[53]  F. Wolfenden,et al.  Developing academics’ assessment practices in open, distance and e-learning: an institutional change agenda , 2016 .

[54]  Anne Jelfs,et al.  Perceptions of academic quality and approaches to studying among disabled students and nondisabled students in distance education , 2009 .

[55]  Bart Rienties,et al.  Assessing Learning Gains , 2016, TEA.

[56]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[57]  John T. E. Richardson,et al.  The attainment of White and ethnic minority students in distance education , 2012 .

[58]  Jude Carroll,et al.  If I was going there I wouldn’t start from here: a critical commentary on current assessment practice , 2011 .

[59]  Gráinne Conole,et al.  Designing for Learning in an Open World , 2012 .

[60]  P. Black,et al.  Meanings and Consequences: a basis for distinguishing formative and summative functions of assessment? , 1996 .

[61]  Bernard Scott,et al.  Competencies for learning design: A review of the literature and a proposed framework , 2011, Br. J. Educ. Technol..

[62]  Julie L. Booth,et al.  Instructional Complexity and the Science to Constrain It , 2013, Science.

[63]  Bart Rienties,et al.  Learning Design – creative design to visualise learning activities , 2016 .

[64]  David S. Haviland,et al.  Designing for Educational Technology. , 1970 .

[65]  Skills Fulfilling our potential: teaching excellence, social mobility and student choice , 2015 .

[66]  Bart Rienties,et al.  "Scaling up" learning design: impact of learning design activities on LMS behavior and performance , 2015, LAK.

[67]  Bart Rienties,et al.  Modelling and Managing Learner Satisfaction: Use of Learner Feedback to Enhance Blended and Online Learning Experience , 2016 .

[68]  Elise Lavoué,et al.  Design for Teaching and Learning in a Networked World: 10th European Conference on Technology Enhanced Learning, EC-TEL 2015, Toledo, Spain, September 15-18, 2015, Proceedings , 2015, EC-TEL.

[69]  Hendrik Drachsler,et al.  Privacy and Analytics – it’s a DELICATE issue. A Checklist to establish trusted Learning Analytics , 2016 .

[70]  Marie Schmidt,et al.  Learning to Teach in Higher Education , 1992 .

[71]  S. A. Becker,et al.  NMC Horizon Report: 2016 Higher Education Edition , 2015 .

[72]  Mantz Yorke,et al.  Enigmatic Variations: Honours Degree Assessment Regulations in the UK. , 2008 .

[73]  Peter Mork,et al.  From Data to Decisions: A Value Chain for Big Data , 2013, IT Professional.

[74]  Bart Rienties,et al.  The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules , 2016, Comput. Hum. Behav..

[75]  Norbert Pachler,et al.  Exploring formative e‐assessment: using case stories and design patterns , 2010 .

[76]  Patrina Law,et al.  Digital badging at The Open University: recognition for informal learning , 2015 .

[77]  Sergey Popov,et al.  University Competition, Grading Standards, and Grade Inflation , 2010 .

[78]  Alan Woodley,et al.  National student feedback surveys in distance education: an investigation at the UK Open University , 2011 .

[79]  Sally Jordan,et al.  Same but different, but is it fair? An analysis of the use of variants of interactive computer-marked questions , 2011 .

[80]  Alison Ashby Monitoring student retention in the Open University: definition, measurement, interpretation and action , 2004 .

[81]  J. T. Richardson The National Student Survey and its impact on UK higher education , 2013 .

[82]  Chris Rust The international state of research on assessment & examinations in Higher Education , 2015 .

[83]  Eric Andres,et al.  Exploring feedback and student characteristics relevant for personalizing feedback strategies , 2014, Comput. Educ..

[84]  Bart Rienties,et al.  Analysing 157 learning designs using learning analytic approaches as a means to evaluate the impact of pedagogical decision making , 2016, Br. J. Educ. Technol..

[85]  John T. E. Richardson,et al.  Academic Attainment of Students with Disabilities in Distance Education. , 2014 .

[86]  Laurie G. Hillstock,et al.  Exploring Characteristics of Retained First-Year Students Enrolled in Non-Proximal Distance Learning Programs , 2014 .

[87]  D. Sluijsmans,et al.  Effective peer assessment processes: Research findings and future directions , 2010 .

[88]  D. Suthers,et al.  Proceedings of the Third International Conference on Learning Analytics and Knowledge , 2013, LAK 2013.

[89]  John T. E. Richardson,et al.  The academic attainment of students with disabilities in UK higher education , 2009 .

[90]  Dale H. Schunk,et al.  Self-Regulated Learning: From Self-Management to Self-Definition. , 2003 .

[91]  Kathleen M. T. Collins,et al.  Students’ Perceptions of Characteristics of Effective College Teachers: A Validity Study of a Teaching Evaluation Form Using a Mixed-Methods Analysis , 2007 .

[92]  Denise Whitelock,et al.  Activating Assessment for Learning: Are We on the Way with Web 2.0? , 2010 .

[93]  Michelle Meadows,et al.  A review of the literature on marking reliability , 2005 .

[94]  Zdenek Zdráhal,et al.  Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment , 2013, LAK '13.

[95]  Rebecca Ferguson,et al.  MOOCS: What The Open University research tells us , 2016 .

[96]  Denise Whitelock,et al.  DOES THE QUALITY AND QUANTITY OF EXAM REVISION IMPACT ON STUDENT SATISFACTION AND PERFORMANCE IN THE EXAM ITSELF?: PERSPECTIVES FROM UNDERGRADUATE DISTANCE LEARNERS , 2016 .

[97]  Svenja Bedenlier,et al.  The battle for open: how openness won and why it doesn’t feel like victory , 2015 .

[98]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[99]  Rebecca Ferguson,et al.  Taking on Different Roles: How Educators Position Themselves in MOOCs , 2014, EC-TEL.

[100]  Yeliz Yesilada,et al.  Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A): Building the mobile web: rediscovering accessibility? , 2006 .

[101]  Carol Calvert,et al.  Developing a model and applications for probabilities of student success: a case study of predictive analytics , 2014 .

[102]  M. Eraut Non-formal learning and tacit knowledge in professional work. , 2000, The British journal of educational psychology.

[103]  John T. E. Richardson,et al.  The under-attainment of ethnic minority students in UK higher education: what we know and what we don’t know , 2015 .

[104]  John T. E. Richardson,et al.  Coursework versus examinations in end-of-module assessment: a literature review , 2015 .

[105]  Denise Whitelock,et al.  Computer assisted formative assessment: supporting students to become more reflective learners , 2007 .

[106]  D. Sadler Formative assessment and the design of instructional systems , 1989 .

[107]  Sue Bennett,et al.  Support for assessment practice: developing the Assessment Design Decisions Framework , 2016 .

[108]  Qing Gu,et al.  Learning and growing in a ‘foreign’ context: intercultural experiences of international students , 2010 .

[109]  Anna Mountford-Zimdars,et al.  Causes of differences in student outcomes , 2015 .

[110]  George D. Kuh Assessing What Really Matters to Student Learning Inside The National Survey of Student Engagement , 2001 .

[111]  Sean B. Eom,et al.  The Determinants of Students’ Perceived Learning Outcomes and Satisfaction in University Online Education: An Update* , 2016 .

[112]  Herbert W. Marsh,et al.  SEEQ: A Reliable, Valid, and Useful Instrument for Collecting Students' Evaluations of University Teaching. , 1982 .

[113]  Justin D. Cochran,et al.  The Role of Student Characteristics in Predicting Retention in Online Courses , 2014 .

[114]  Jane Seale,et al.  Not the right kind of 'digital capital'? An examination of the complex relationship between disabled students, their technologies and higher education institutions , 2015, Comput. Educ..

[115]  Ainsworth Donovan,et al.  Recognition of non-formal and informal learning , 2014 .

[116]  Sarah J. Stein,et al.  Can you increase teacher engagement with evaluation simply by improving the evaluation system? , 2016 .

[117]  What 'retention' means to me: the position of the adult learner in student retention , 2013 .

[118]  George D. Kuh,et al.  Being (Dis)Engaged in Educationally Purposeful Activities: The Influences of Student and Institutional Characteristics , 2002 .

[119]  Alejandro Armellini,et al.  Learning design and assessment with e-tivities , 2010, Br. J. Educ. Technol..