Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success

Abstract This study examined the extent to which instructional conditions influence the prediction of academic success in nine undergraduate courses offered in a blended learning model (n = 4134). The study illustrates the differences in predictive power and significant predictors between course-specific models and generalized predictive models. The results suggest that it is imperative for learning analytics research to account for the diverse ways technology is adopted and applied in course-specific contexts. The differences in technology use, especially those related to whether and how learners use the learning management system, require consideration before the log-data can be merged to create a generalized model for predicting academic success. A lack of attention to instructional conditions can lead to an over or under estimation of the effects of LMS features on students' academic success. These findings have broader implications for institutions seeking generalized and portable models for identifying students at risk of academic failure.

[1]  Ethelene Whitmire,et al.  Disciplinary differences and undergraduates' information-seeking behavior , 2002, J. Assoc. Inf. Sci. Technol..

[2]  R. Sawyer.  The Cambridge Handbook of the Learning Sciences: Introduction , 2014 .

[3]  Catherine L. Finnegan,et al.  Differences by Course Discipline on Student Behavior, Persistence, and Achievement in Online Courses of Undergraduate General Education , 2008 .

[4]  Philip H. Winne,et al.  A METACOGNITIVE VIEW OF INDIVIDUAL DIFFERENCES IN SELF-REGULATED LEARNING , 1996 .

[5]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[6]  Maria Bannert,et al.  e-Research and learning theory: What do sequence and process mining methods contribute? , 2014, Br. J. Educ. Technol..

[7]  Eitel J. M. Lauría,et al.  Mining academic data to improve college student retention: an open source perspective , 2012, LAK.

[8]  D. Perkins.  The Fingertip Effect: How Information-Processing Technology Shapes Thinking , 1985 .

[9]  Kinshuk,et al.  Guest Editorial - Technology Supported Cognition and Exploratory Learning , 2012, J. Educ. Technol. Soc..

[10]  Dragan Gasevic,et al.  Analytics of the effects of video use and instruction to support reflective learning , 2014, LAK.

[11]  Y. Engeström,et al.  Learning by expanding: An activity-theoretical approach to developmental research , 2014 .

[12]  R. Bjork,et al.  Self-regulated learning: beliefs, techniques, and illusions. , 2013, Annual review of psychology.

[13]  E. Deci,et al.  The effects of instructors' autonomy support and students' autonomous motivation on learning organic chemistry: A self-determination theory perspective , 2000 .

[14]  Etienne Wenger,et al.  Situated Learning: Legitimate Peripheral Participation , 1991 .

[15]  Geraldine Clarebout,et al.  Content Management Systems: Enriched learning opportunities for all? , 2012, Comput. Hum. Behav..

[16]  Charles R. Graham.  BLENDED LEARNING SYSTEMS: DEFINITION, CURRENT TRENDS, AND FUTURE DIRECTIONS , 2005 .

[17]  Sebastián Ventura,et al.  Data mining in course management systems: Moodle case study and tutorial , 2008, Comput. Educ..

[18]  Shane Dawson,et al.  Informing Pedagogical Action , 2013 .

[19]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[20]  Mike Sharkey,et al.  Course correction: using analytics to predict course success , 2012, LAK '12.

[21]  Marek Hatala,et al.  Social presence in online discussions as a process predictor of academic performance , 2015, J. Comput. Assist. Learn..

[22]  Daniel C. Moos,et al.  Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? , 2008 .

[23]  Z. Kovacic,et al.  Predicting student success by mining enrolment data. , 2012 .

[24]  A. Wise,et al.  Broadening the notion of participation in online discussions: examining patterns in learners’ online listening behaviors , 2013 .

[25]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[26]  Graham J Hole,et al.  How to Design and Report Experiments , 2002 .

[27]  Geraldine Clarebout,et al.  Metacognition and the use of tools , 2013 .

[28]  Stuart Palmer,et al.  Modelling Engineering Student Academic Performance Using Academic Analytics , 2013 .

[29]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[30]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[31]  Marek Hatala,et al.  Factors influencing beliefs for adoption of a learning analytics tool: An empirical study , 2013, Comput. Educ..

[32]  Karen Swan,et al.  Linking online course design and implementation to learning outcomes: A design experiment , 2012, Internet High. Educ..

[33]  Alberto Salguero,et al.  Factors influencing university drop out rates , 2009, Comput. Educ..

[34]  Ryan S. Baker,et al.  Educational Data Mining and Learning Analytics , 2014 .

[35]  Philip H. Winne,et al.  Studying as self-regulated learning. , 1998 .

[36]  D. Randy Garrison,et al.  Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education , 1999, Internet High. Educ..

[37]  Sebastián Ventura,et al.  Predicting students' final performance from participation in on-line discussion forums , 2013, Comput. Educ..

[38]  John P. Campbell,et al.  Academic Analytics: A New Tool for a New Era. , 2007 .

[39]  K. Trigwell,et al.  Relations between teachers' approaches to teaching and students' approaches to learning , 1999 .

[40]  Shane Dawson,et al.  Numbers Are Not Enough. Why e-Learning Analytics Failed to Inform an Institutional Strategic Plan , 2012, J. Educ. Technol. Soc..

[41]  Philip H. Winne,et al.  Minimizing the black box problem to enhance the validity of theories about instructional effects , 1982 .

[42]  Jay Liebowitz,et al.  Facilitating Innovation through Knowledge Sharing: A Look at the Us Naval Surface Warfare Center-Carderock Division , 2002, J. Comput. Inf. Syst..

[43]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[44]  Leah P. Macfadyen,et al.  Embracing Big Data in Complex Educational Systems: The Learning Analytics Imperative and the Policy Challenge. , 2014 .

[45]  George Siemens,et al.  Penetrating the fog: analytics in learning and education , 2014 .

[46]  Philip H. Winne,et al.  How Software Technologies Can Improve Research on Learning and Bolster School Reform , 2006 .

[47]  Eitel J. M. Lauría,et al.  Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative , 2014, J. Learn. Anal..

[48]  Miguel Ángel Conde González,et al.  Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning , 2014, Comput. Hum. Behav..

[49]  Geraldine Clarebout,et al.  Tool-use in a blended undergraduate course: In Search of user profiles , 2011, Comput. Educ..

[50]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[51]  Geraldine Clarebout,et al.  Regulation of tool-use within a blended course: Student differences and performance effects , 2013, Comput. Educ..

[52]  George Siemens,et al.  Let’s not forget: Learning analytics are about learning , 2015 .

[53]  Marek Hatala,et al.  Automated Cognitive Presence Detection in Online Discussion Transcripts , 2014, LAK Workshops.

[54]  Sock H. Chung,et al.  An Empirical Study of Students' Computer Self-Efficacy: Differences among Four Academic Disciplines at a Large University , 2002, J. Comput. Inf. Syst..

[55]  O. Keene,et al.  The log transformation is special. , 1995, Statistics in medicine.

[56]  Moon-Heum Cho,et al.  Students' self-regulation for interaction with others in online learning environments , 2013, Internet High. Educ..

[57]  Peter Goodyear,et al.  Learning through face-to-face and online discussions: Associations between students' conceptions, approaches and academic performance in political science , 2010, Br. J. Educ. Technol..

[58]  George Siemens,et al.  Guest Editorial - Learning and Knowledge Analytics , 2012, J. Educ. Technol. Soc..

[59]  Stuart R. Palmer,et al.  Does the discussion help? The impact of a formally assessed online discussion on final student results , 2008, Br. J. Educ. Technol..

[60]  Jeffrey W. Alstete,et al.  Performance indicators in online distance learning courses: a study of management education , 2004 .

[61]  Philip H. Winne,et al.  A Cognitive and Metacognitive Analysis of Self-Regulated Learning , 2011 .

[62]  George Siemens,et al.  Current state and future trends: a citation network analysis of the learning analytics field , 2014, LAK.

[63]  Dragan Gasevic,et al.  Learning Analytics and the Imperative for Theory Driven Research , 2016 .

[64]  B. Zimmerman,et al.  Handbook of Self-Regulation of Learning and Performance , 2011 .

[65]  Sebastián Ventura,et al.  Multiple instance learning for classifying students in learning management systems , 2011, Expert Syst. Appl..

[66]  D. Garrison,et al.  Facilitating Cognitive Presence in Online Learning: Interaction Is Not Enough , 2005 .

[67]  R. H. Myers.  Classical and modern regression with applications , 1986 .

[68]  R. Ellis,et al.  Quality in blended learning: Exploring the relationships between on-line and face-to-face teaching and learning , 2007, Internet High. Educ..