Measuring student engagement in technology-mediated learning: A review

Using digital technology to deliver content, connect learners, and enable anytime, anywhere learning is increasing, but keeping students engaged in technology-mediated learning is challenging. Instructional practices that encourage greater engagement are essential if we are to effectively use digital instructional technologies. To determine the impact of innovative instructional practices on learning, we need useful measures of student engagement. These measures should be adaptable to the unique challenges to studying technology-mediated learning, such as when students learn at a distance or in a blended learning course. In this review, we examine existing approaches to measure engagement in technology-mediated learning. We identify strengths and limitations of existing measures and outline potential approaches to improve the measurement of student engagement. Our intent is to assist researchers, instructors, designers, and others in identifying effective methods to conceptualize and measure student engagement in technology-mediated learning. Consensus is needed for the definition and operationalization of student engagement.Most technology-mediated learning research uses self-report measures of engagement.Physiological and systems data offer an alternative method to measuring engagement.More research is needed to study the role of emotional engagement in learning.More research needed to determine value of physiological and systems data.

[1]  Kerry Rice,et al.  A Comprehensive Look at Distance Education in the K–12 Context , 2006 .

[2]  Vincent Tinto.  Dropout from Higher Education: A Theoretical Synthesis of Recent Research , 1975 .

[3]  Janet Jane Fiona Martin,et al.  Designing for Users: Online Information Literacy in the Middle East , 2009 .

[4]  T. Kindermann,et al.  Engagement and Disaffection in the Classroom: Part of a Larger Motivational Dynamic? , 2008 .

[5]  B. M. Fulk,et al.  Comparing Online and Face-to-Face Presentation of Course Content in an Introductory Special Education Course , 2012 .

[6]  B. Jean Mandernach,et al.  Journals for Computer-Mediated Learning: Publications of Value for the Online Educator , 2009 .

[7]  Sanna Järvelä,et al.  Face-to-face encounters as contextual support for Web-based discussions in a teacher education course , 2004, Internet High. Educ..

[8]  Ernest T. Pascarella,et al.  An Assessment of the Academic and Social Influences on Freshman Year Educational Outcomes , 2017 .

[9]  P. David Pearson.  Handbook of reading research. , 1990 .

[10]  William J. Hussar,et al.  The Condition of Education 2012. NCES 2012-045. , 2012 .

[11]  E. Skinner,et al.  Developmental Dynamics of Student Engagement, Coping, and Everyday Resilience , 2012 .

[12]  Yaoying Xu,et al.  Examining the Effects of Digital Feedback on Student Engagement and Achievement , 2010 .

[13]  Heather Staker,et al.  The Rise of K-12 Blended Learning: Profiles of Emerging Models. , 2011 .

[14]  Charles R. Graham,et al.  Exploring Intensive Longitudinal Measures of Student Engagement in Blended Learning , 2015 .

[15]  Minjuan Wang,et al.  Affective e-Learning: Using "Emotional" Data to Improve Learning in Pervasive Learning Environment , 2009, J. Educ. Technol. Soc..

[16]  H. M. Marks,et al.  Student Engagement in Instructional Activity: Patterns in the Elementary, Middle, and High School Years , 2000 .

[17]  James J. Appleton,et al.  Student engagement with school: Critical conceptual and methodological issues of the construct. , 2008 .

[18]  Kennon M. Sheldon,et al.  Teacher support, student motivation, student need satisfaction, and college teacher course evaluations: testing a sequential path model , 2008 .

[19]  Cher Ping Lim,et al.  Gaming in a 3D multiuser virtual environment: engaging students in Science lessons , 2006, Br. J. Educ. Technol..

[20]  John M. Beckem,et al.  BRINGING LIFE TO LEARNING: IMMERSIVE EXPERIENTIAL LEARNING SIMULATIONS FOR ONLINE AND BLENDED COURSES , 2012 .

[21]  Irshat Madyarov,et al.  Designing a Workable Framework for Evaluating Distance Language Instruction , 2013 .

[22]  Katherine Schilling,et al.  The Impact of Multimedia Course Enhancements on Student Learning Outcomes , 2009 .

[23]  Mei-jung Wang,et al.  Online collaboration and offline interaction between students using asynchronous tools in blended learning , 2010 .

[24]  Candace Figg,et al.  Exploring teacher knowledge and actions supporting technology-enhanced teaching in elementary schools: Two approaches by pre-service teachers , 2011 .

[25]  Beverly Park Woolf,et al.  Affect-aware tutors: recognising and responding to student affect , 2009, Int. J. Learn. Technol..

[26]  A. Wise,et al.  The Effects of Teacher Social Presence on Student Satisfaction, Engagement, and Learning , 2004, ICLS.

[27]  Deborah Miller,et al.  Engaging college science students and changing academic achievement with technology: A quasi-experimental preliminary investigation , 2009, Comput. Educ..

[28]  G. Humiston,et al.  A Comparative Analysis of Student Engagement, Learning, and Satisfaction in Lecture Hall and Online Learning Settings. , 2009 .

[29]  Robert Rueda,et al.  Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education , 2012, Br. J. Educ. Technol..

[30]  Damian Bebell,et al.  One to One Computing: A Summary of the Quantitative Results from the Berkshire Wireless Learning Initiative , 2010 .

[31]  Dianna J. Spence,et al.  Engagement with Mathematics Courseware in Traditional and Online Remedial Learning Environments: Relationship to Self-Efficacy and Achievement , 2007 .

[32]  Beverly Park Woolf,et al.  Inferring Unobservable Learning Variables from Students' Help Seeking Behavior , 2004, Intelligent Tutoring Systems.

[33]  Heeyoung Kim,et al.  Just-in-Time Teaching Exercises to Engage Students in an Introductory-Level Dinosaur Course , 2007 .

[34]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[35]  Carolyn M. Callahan,et al.  Describing Learning in an Advanced Online Case-Based Course in Environmental Science , 2010 .

[36]  Mary Jane White,et al.  Teacher Interaction: Motivating At-Risk Students in Web-based High School Courses , 2001 .

[37]  G. Ladd,et al.  Continuity and Change in Early School Engagement: Predictive of Children's Achievement Trajectories from First to Eighth Grade? , 2009, Journal of educational psychology.

[38]  Robert L. Bangert-Drowns,et al.  A Taxonomy of Student Engagement with Educational Software: An Exploration of Literate Thinking with Electronic Text , 2001 .

[39]  Anne-Marie Nuttall,et al.  Student Engagement Patterns over the Duration of Level 1 and Level 3 Geography Modules: Influences on Student Attendance, Performance and Use of Online Resources , 2011 .

[40]  Charles R. Graham,et al.  An analysis of high impact scholarship and publication trends in blended learning , 2012 .

[41]  S. Barab,et al.  The STAR Project: Enhancing Adolescents' Social Understanding through Video-Based, Multimedia Scenarios , 2000 .

[42]  Vincent Aleven,et al.  Sensor-free automated detection of affect in a Cognitive Tutor for Algebra , 2012, EDM.

[43]  John T. E. Richardson,et al.  Academic engagement in students with a hearing loss in distance education. , 2004, Journal of deaf studies and deaf education.

[44]  Tina L. Heafner,et al.  Wikis and Constructivism in Secondary Social Studies: Fostering a Deeper Understanding , 2008 .

[45]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[46]  C. C. Robinson,et al.  New Benchmarks in Higher Education: Student Engagement in Online Learning , 2008 .

[47]  T. Kindermann,et al.  A Motivational Perspective on Engagement and Disaffection , 2009 .

[48]  Doris U. Bolliger,et al.  Active learning in the online environment: The integration of student-generated audio files , 2013 .

[49]  A. Wise,et al.  Broadening the notion of participation in online discussions: examining patterns in learners’ online listening behaviors , 2013 .

[50]  L. Morris,et al.  Tracking student behavior, persistence, and achievement in online courses , 2005, Internet High. Educ..

[51]  Robert L. Bangert-Drowns,et al.  Teacher ratings of student engagement with educational software: An exploratory study , 2002 .

[52]  Robin H. Kay,et al.  Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale , 2011, Comput. Hum. Behav..

[53]  Arthur C. Graesser,et al.  AutoTutor and affective autotutor: Learning by talking with cognitively and emotionally intelligent computers that talk back , 2012, TIIS.

[54]  K. Moore,et al.  What do children need to flourish? : conceptualizing and measuring indicators of positive development , 2005 .

[55]  Robert Eckert,et al.  The Effects of Microcomputers on the Secondary Special Education Classroom Ecology , 1987 .

[56]  Bryan Hosack,et al.  A Framework for Measuring Student Learning Gains and Engagement in an Introductory Computing Course: A Preliminary Report of Findings , 2012 .

[57]  Karen Swan,et al.  Examining the extent and nature of online learning in American K-12 Education: The research initiatives of the Alfred P. Sloan Foundation , 2012, Internet High. Educ..

[58]  Joel M. Hektner,et al.  Experience sampling method , 2007 .

[59]  Siu Cheung Kong,et al.  An evaluation study of the use of a cognitive tool in a one-to-one classroom for promoting classroom-based dialogic interaction , 2011, Comput. Educ..

[60]  R. Eklund,et al.  The Flow Scales Manual , 2004 .

[61]  William J. Miller,et al.  Classroom as Reality: Demonstrating Campaign Effects Through Live Simulation , 2011 .

[62]  Sarah Nakamaru,et al.  Investment and Return , 2011 .

[63]  Vincent Aleven,et al.  Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra. , 2012, EDM 2012.

[64]  Erping Zhu,et al.  Interaction and cognitive engagement: An analysis of four asynchronous online discussions , 2006 .

[65]  Robert M. Gonyea,et al.  Unmasking the Effects of Student Engagement on First-Year College Grades and Persistence , 2008 .

[66]  Alec M. Bodzin,et al.  A mixed methods assessment of students' flow experiences during a mobile augmented reality science game , 2013, J. Comput. Assist. Learn..

[67]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[68]  Marie Bienkowski,et al.  Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief , 2012 .

[69]  Kendall Hartley,et al.  An Examination of One-to-One Computing in the Middle School: Does Increased Access Bring about Increased Student Engagement? , 2010 .

[70]  Dianne Conrad,et al.  Engagement, Excitement, Anxiety, and Fear: Learners' Experiences of Starting an Online Course , 2002 .

[71]  Hamish Coates,et al.  A model of online and general campus‐based student engagement , 2007 .

[72]  B. Patterson,et al.  Attrition in Online and Campus Degree Programs. , 2009 .

[73]  K. Hayden,et al.  Increasing Student Interest and Attitudes in STEM: Professional Development and Activities to Engage and Inspire Learners , 2011 .

[74]  B. Jean Mandernach.  Effect of Instructor-Personalized Multimedia in the Online Classroom , 2009 .

[75]  Wendy McColskey,et al.  The Measurement of Student Engagement: A Comparative Analysis of Various Methods and Student Self-report Instruments , 2012 .

[76]  Ron Owston,et al.  Student perceptions and achievement in a university blended learning strategic initiative , 2013, Internet High. Educ..

[77]  Lisa Linnenbrink-Garcia,et al.  Academic Emotions and Student Engagement , 2012 .

[78]  S. Christenson,et al.  Jingle, Jangle, 1 and Conceptual Haziness 2 : Evolution and Future Directions of the Engagement Construct , 2012 .

[79]  Steven P. Cole,et al.  Teacher assisted versus individual viewing of foreign language video: Relation to comprehension, self-efficacy, and engagement , 2013 .

[80]  Michel Janosz,et al.  Part IV commentary: Outcomes of engagement and engagement as an outcome: Some consensus, divergences, and unanswered questions. , 2012 .

[81]  Melissa Erlene Pierson,et al.  Technology Integration Practice as a Function of Pedagogical Expertise , 2001 .

[82]  John T. E. Richardson,et al.  Academic Engagement and Perceptions of Quality in Distance Education , 2003 .

[83]  Robert M. Gonyea,et al.  Learning at a Distance: Engaged or Not? , 2008 .

[84]  Herbert J. Rieth,et al.  An Analysis of the Impact of Microcomputers on the Secondary Special Education Classroom Ecology , 1988 .

[85]  Liesel Knaack,et al.  A formative analysis of individual differences in the effectiveness of learning objects in secondary school , 2008, Comput. Educ..

[86]  Jennifer A. Fredricks,et al.  Measuring student engagement in upper elementary through high school : a description of 21 instruments January 2011 Prepared , 2010 .

[87]  Joseph B. Berger,et al.  THE ROLE OF STUDENT INVOLVEMENT AND PERCEPTIONS OF INTEGRATION IN A CAUSAL MODEL OF STUDENT PERSISTENCE , 1999 .

[88]  Robert A. Smallwood,et al.  Assessment Measures: CLASSE--The Class-Level Survey of Student Engagement. , 2005 .

[89]  Jim Gleason,et al.  Using Technology-Assisted Instruction and Assessment to Reduce the Effect of Class Size on Student Outcomes in Undergraduate Mathematics Courses , 2012 .

[90]  Arthur C. Graesser,et al.  Toward an Affect-Sensitive AutoTutor , 2007, IEEE Intelligent Systems.

[91]  M. D. Roblyer,et al.  Virtually Successful: Defeating the Dropout Problem through Online School Programs , 2006 .

[92]  Synnöve Kekkonen-Moneta,et al.  Affective Learning in Online Multimedia and Lecture Versions of an Introductory Computing Course , 2007 .

[93]  Curtis J. Bonk,et al.  DOES SENSE OF COMMUNITY MATTER? an Examination of Participants' Perceptions of Building Learning Communities in Online Courses , 2007 .

[94]  Peter G. Cole,et al.  Teaching Principles and Practice , 1994 .

[95]  Xiaotian Chen.  Google Scholar's Dramatic Coverage Improvement Five Years after Debut 1 1 The author is grateful to Alice Chen of Duke University for editing this paper. , 2010 .

[96]  Robin Kay,et al.  Assessing learning, quality and engagement in learning objects: the Learning Object Evaluation Scale for Students (LOES-S) , 2009 .

[97]  Hamish Coates,et al.  Student Engagement in Campus-Based and Online Education: University Connections , 2006 .

[98]  R. Baker,et al.  Operationalizing and Detecting Disengagement Within Online Science Microworlds , 2015 .

[99]  Allan Wigfield,et al.  Engagement and Motivation in Reading , 2000 .

[100]  George D. Kuh.  Assessing What Really Matters to Student Learning Inside The National Survey of Student Engagement , 2001 .

[101]  Joseph Betts,et al.  Issues and Methods in the Measurement of Student Engagement: Advancing the Construct Through Statistical Modeling , 2012 .

[102]  Kasia Muldner,et al.  The Effect of Motivational Learning Companions on Low Achieving Students and Students with Disabilities , 2010, Intelligent Tutoring Systems.

[103]  Kevin R. Guidry,et al.  Engaging online learners: The impact of Web-based learning technology on college student engagement , 2010, Comput. Educ..

[104]  Jonathan P. Rowe,et al.  Integrating Learning, Problem Solving, and Engagement in Narrative-Centered Learning Environments , 2011, Int. J. Artif. Intell. Educ..

[105]  Vincent Aleven,et al.  Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor , 2006, Int. J. Artif. Intell. Educ..

[106]  Sanna Järvelä,et al.  Investigating student engagement in computer-supported inquiry: a process-oriented analysis , 2008 .

[107]  Dirk T. Tempelaar,et al.  A dynamic analysis of the interplay between asynchronous and synchronous communication in online learning: The impact of motivation , 2014, J. Comput. Assist. Learn..

[108]  Katy Jordan,et al.  Initial trends in enrolment and completion of massive open online courses , 2014 .

[109]  Hai-Ning Liang,et al.  Can interactive visualization tools engage and support pre-university students in exploring non-trivial mathematical concepts? , 2010, Comput. Educ..

[110]  D'melloSidney,et al.  AutoTutor and affective autotutor , 2013 .

[111]  John F. Watson,et al.  Keeping Pace with K-12 Digital Learning: An Annual Review of Policy and Practice. Eleventh Edition. , 2014 .

[112]  Namin Shin,et al.  Online learner's 'flow' experience: an empirical study , 2006, Br. J. Educ. Technol..

[113]  Carina Granberg,et al.  Social software for reflective dialogue: questions about reflection and dialogue in student teachers’ blogs , 2010 .

[114]  Shirley Shmerling,et al.  Constructivist Pedagogy in Asynchronous Online Education: Examining Proactive Behavior and the Impact on Student Engagement Levels , 2011 .

[115]  George Siemens,et al.  Penetrating the fog: analytics in learning and education , 2014 .

[116]  Jean-Michel Boucheix,et al.  Cueing animations: Dynamic signaling aids information extraction and comprehension , 2013 .

[117]  Daniel D. Barron,et al.  Participants’ Perceptions of Web-Infused Environments , 2001 .

[118]  Mihaela Cocea,et al.  Disengagement Detection in Online Learning: Validation Studies and Perspectives , 2011, IEEE Transactions on Learning Technologies.

[119]  D. Watson,et al.  Development and validation of brief measures of positive and negative affect: the PANAS scales. , 1988, Journal of personality and social psychology.

[120]  Ari Korhonen,et al.  Comparing Learning Performance of Students Using Algorithm Visualizations Collaboratively on Different Engagement Levels , 2009, J. Educ. Technol. Soc..

[121]  A. A. G. Putra,et al.  PIECING TOGETHER THE STUDENT SUCCESS PUZZLE: RESEARCH, PROPOSITIONS, AND RECOMMENDATIONS , 2013 .

[122]  Gregory Schraw,et al.  Situational Interest in Literary Text , 1997, Contemporary educational psychology.

[123]  Reynol Junco,et al.  The effect of Twitter on college student engagement and grades , 2011, J. Comput. Assist. Learn..

[124]  Anastasia Kitsantas,et al.  Developmental phases in self-regulation : Shifting from process goals to outcome goals , 1997 .

[125]  Adam Finkelstein,et al.  Understanding the effects of professors' pedagogical development with Clicker Assessment and Feedback technologies and the impact on students' engagement and learning in higher education , 2013, Comput. Educ..

[126]  Brian Horvitz,et al.  ‘Isn’t It Just Like Our Situation?’ Engagement and Learning in an Online Story-Based Environment , 2006 .

[127]  David Lester Neumann,et al.  The Effects of Using a Wiki on Student Engagement and Learning of Report Writing Skills in a University Statistics Course. , 2009 .

[128]  Kasia Muldner,et al.  Affective Tutors: Automatic Detection of and Response to Student Emotion , 2010, Advances in Intelligent Tutoring Systems.

[129]  Yu-Fen Yang,et al.  Engaging students in an online situated language learning environment , 2011 .

[130]  John T. E. Richardson,et al.  Face‐to‐face versus online tutoring support in distance education , 2007 .

[131]  Cher Ping Lim.  Global citizenship education, school curriculum and games: Learning Mathematics, English and Science as a global citizen , 2008, Comput. Educ..

[132]  K. Samuelsen,et al.  Part V Commentary: Possible New Directions in the Measurement of Student Engagement , 2012 .

[133]  Kaye Stacey,et al.  A scale for monitoring students' attitudes to learning mathematics with technology , 2007, Comput. Educ..

[134]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[135]  Angela B. Garcia,et al.  Effects of Student Control and Choice on Engagement in a CAI Arithmetic Task in a Low-Income School. , 1975 .

[136]  Dc Washington,et al.  Distance education at degree-granting postsecondary institutions: (NCES -044). U.S. Department of Education, Institute of Education for Education Statistics. , 2009 .

[137]  Carmel McNaught,et al.  Investigating relationships between features of learning designs and student learning outcomes , 2012 .

[138]  Lorena Blasco-Arcas,et al.  Using clickers in class. The role of interactivity, active collaborative learning and engagement in learning performance , 2013, Comput. Educ..

[139]  Reinhard Pekrun,et al.  Emotions as Drivers of Learning and Cognitive Development , 2011 .

[140]  Kay McClenney,et al.  The Effort—Outcome Gap: Differences for African American and Hispanic Community College Students in Student Engagement and Academic Achievement , 2008 .

[141]  James J. Appleton,et al.  Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument , 2006 .

[142]  A. Astin.  Student involvement: A developmental theory for higher education. , 1999 .

[143]  Robin Kay,et al.  Evaluating the learning in learning objects , 2007 .

[144]  Marcia D. Dixson.  Creating effective student engagement in online courses: What do students find engaging? , 2010 .

[145]  M. Csíkszentmihályi,et al.  Experience Sampling Method: Measuring the Quality of Everyday Life , 2006 .

[146]  J. Hughes,et al.  Teacher-Student Support, Effortful Engagement, and Achievement: A 3-Year Longitudinal Study. , 2008, Journal of educational psychology.

[147]  Sarah K. Howard,et al.  Professional identity creation: examining the development of beginning preservice teachers¿ understanding of their work as teachers , 2010 .

[148]  Anastasios Barkatsas,et al.  Learning secondary mathematics with technology: Exploring the complex interrelationship between students' attitudes, engagement, gender and achievement , 2009, Comput. Educ..

[149]  W. Briggs,et al.  A Measure of College Student Course Engagement , 2005 .