Measuring Student Engagement in Technology-Mediated Learning Environments.

Measuring Student Engagement in Technology-Mediated Learning Environments Curtis R. Henrie Department of Instructional Psychology and Technology, BYU Doctor of Philosophy This is a multiple-article format dissertation that explores methods for measuring student engagement in technology-mediated learning experiences. Student engagement is the committed, focused, and energetic involvement of students in learning. Student engagement is correlated with academic performance, student satisfaction, and persistence in learning, making it a valuable predictor of important learning outcomes. In order to identify which students need help or to evaluate how well an instructional interaction promotes student engagement, we need effective measures of student engagement. These measures should be scalable, cost effective, and minimally disruptive to learning. This dissertation examines different approaches to measure student engagement in technology-mediated learning environments that meet the identified measurement criteria. The first article is an extended literature review that examines how engagement has been measured in technology-mediated learning experiences. The second article is an instrument evaluation of an activity-level self-report measure of student engagement. The third article explores the relationships between learning management system user-activity data (log data) and results of the activity-level self-report measure of student engagement.

[1]  Michel Janosz,et al.  Part IV commentary: Outcomes of engagement and engagement as an outcome: Some consensus, divergences, and unanswered questions. , 2012 .

[2]  Ming-Te Wang,et al.  Part I Commentary: So What Is Student Engagement Anyway? , 2012 .

[3]  David J. Shernoff,et al.  Student engagement in high school classrooms from the perspective of flow theory. , 2003 .

[4]  Charles R. Graham,et al.  Measuring student engagement in technology-mediated learning: A review , 2015, Comput. Educ..

[5]  Wendy McColskey,et al.  The Measurement of Student Engagement: A Comparative Analysis of Various Methods and Student Self-report Instruments , 2012 .

[6]  M. Tremblay,et al.  A comparison of direct versus self-report measures for assessing physical activity in adults: a systematic review , 2008, The international journal of behavioral nutrition and physical activity.

[7]  P. Pintrich,et al.  Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq) , 1993 .

[8]  Hasan Çakir,et al.  Research in online learning environments: Priorities and methodologies , 2011, Comput. Educ..

[9]  J. Finn,et al.  Student engagement: What is it? Why does it matter? , 2012 .

[10]  Marcia D. Dixson Creating effective student engagement in online courses: What do students find engaging? , 2010 .

[11]  J. Biggs,et al.  The revised two-factor Study Process Questionnaire: R-SPQ-2F. , 2001, The British journal of educational psychology.

[12]  K. Vohs,et al.  How Emotion Shapes Behavior: Feedback, Anticipation, and Reflection, Rather Than Direct Causation , 2007, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[13]  Thomas M. Smith,et al.  The Condition of education , 1975 .

[14]  Heather Staker,et al.  The Rise of K-12 Blended Learning: Profiles of Emerging Models. , 2011 .

[15]  Rebecca Ferguson,et al.  Learning analytics: drivers, developments and challenges , 2012 .

[16]  S. Christenson,et al.  Jingle, Jangle, 1 and Conceptual Haziness 2 : Evolution and Future Directions of the Engagement Construct , 2012 .

[17]  D'melloSidney,et al.  AutoTutor and affective autotutor , 2013 .

[18]  L. Morris,et al.  Tracking student behavior, persistence, and achievement in online courses , 2005, Internet High. Educ..

[19]  John F. Watson,et al.  Keeping Pace with K-12 Digital Learning: An Annual Review of Policy and Practice. Eleventh Edition. , 2014 .

[20]  Jeffrey A. Owings,et al.  The Adult Lives of At-Risk Students: The Roles of Attainment and Engagement in High School. Statistical Analysis Report. NCES 2006-328. , 2006 .

[21]  E. Skinner,et al.  Developmental Dynamics of Student Engagement, Coping, and Everyday Resilience , 2012 .

[22]  S. Christenson,et al.  Engaging students at school and with learning: A relevant construct for all students , 2008 .

[23]  A. Satorra,et al.  Complex Sample Data in Structural Equation Modeling , 1995 .

[24]  Charles R. Graham,et al.  Exploring Intensive Longitudinal Measures of Student Engagement in Blended Learning , 2015 .

[25]  Kennon M. Sheldon,et al.  Teacher support, student motivation, student need satisfaction, and college teacher course evaluations: testing a sequential path model , 2008 .

[26]  A. A. G. Putra,et al.  PIECING TOGETHER THE STUDENT SUCCESS PUZZLE: RESEARCH, PROPOSITIONS, AND RECOMMENDATIONS , 2013 .

[27]  Jaideep Srivastava,et al.  Data Preparation for Mining World Wide Web Browsing Patterns , 1999, Knowledge and Information Systems.

[28]  K. Moore,et al.  What do children need to flourish? : conceptualizing and measuring indicators of positive development , 2005 .

[29]  Benjamin C. Heddy,et al.  The Challenges of Defining and Measuring Student Engagement in Science , 2015 .

[30]  Randall S. Davies,et al.  The Nature of Parental Interactions in an Online Charter School , 2013 .

[31]  Kyle Kaalberg Enhancing Teaching and Learning , 2014 .

[32]  Jean Donham,et al.  Enhancing teaching and learning , 1998 .

[33]  T. Kindermann,et al.  A Motivational Perspective on Engagement and Disaffection , 2009 .

[34]  Cathy Hammond,et al.  Dropout Risk Factors and Exemplary Programs: A Technical Report. , 2007 .

[35]  Zachary A. Pardos,et al.  Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes , 2013, LAK '13.

[36]  Vincent Aleven,et al.  Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra. , 2012, EDM 2012.

[37]  B. Patterson,et al.  Attrition in Online and Campus Degree Programs. , 2009 .

[38]  M. D. Roblyer,et al.  Virtually Successful: Defeating the Dropout Problem through Online School Programs , 2006 .

[39]  Anastasia Kitsantas,et al.  Developmental phases in self-regulation : Shifting from process goals to outcome goals , 1997 .

[40]  Robert A. Smallwood,et al.  Assessment Measures: CLASSE--The Class-Level Survey of Student Engagement. , 2005 .

[41]  M. Csíkszentmihályi,et al.  Experience Sampling Method: Measuring the Quality of Everyday Life , 2006 .

[42]  M. Heinemann,et al.  Teacher-Student Interaction and Learning In Online Theological Education. Part II: Additional Theoretical Frameworks , 2005 .

[43]  S. Holloway,et al.  What Makes Students Engaged in Learning? A Time-Use Study of Within- and Between-Individual Predictors of Emotional Engagement in Low-Performing High Schools , 2011, Journal of Youth and Adolescence.

[44]  M. Csíkszentmihályi,et al.  Experience Sampling Method Applications to Communication Research Questions. , 1996 .

[45]  J. Reeve,et al.  Agency as a fourth aspect of students engagement during learning activities , 2011 .

[46]  Minjuan Wang,et al.  Affective e-Learning: Using "Emotional" Data to Improve Learning in Pervasive Learning Environment , 2009, J. Educ. Technol. Soc..

[47]  Mihaela Cocea,et al.  Disengagement Detection in Online Learning: Validation Studies and Perspectives , 2011, IEEE Transactions on Learning Technologies.

[48]  Karen Swan,et al.  Examining the extent and nature of online learning in American K-12 Education: The research initiatives of the Alfred P. Sloan Foundation , 2012, Internet High. Educ..

[49]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[50]  J. Hughes,et al.  Teacher-Student Support, Effortful Engagement, and Achievement: A 3-Year Longitudinal Study. , 2008, Journal of educational psychology.

[51]  M. Heinemann Teacher−Student Interaction and Learning in Online Theological Education. Part Four: Findings and Conclusions , 2007 .

[52]  D. Bergin,et al.  Measuring engagement in fourth to twelfth grade classrooms: the Classroom Engagement Inventory. , 2014, School psychology quarterly : the official journal of the Division of School Psychology, American Psychological Association.

[53]  Sidney D'Mello,et al.  The half-life of cognitive-affective states during complex learning , 2011, Cognition & emotion.

[54]  Jan C. Frijters,et al.  A Review of Self-Report and Alternative Approaches in the Measurement of Student Motivation , 2009 .

[55]  Gleb Gusev,et al.  Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments , 2015, WWW.

[56]  M. Csíkszentmihályi,et al.  Validity and Reliability of the Experience‐Sampling Method , 1987, The Journal of nervous and mental disease.

[57]  Hal A. Lawson,et al.  New Conceptual Frameworks for Student Engagement Research, Policy, and Practice , 2013 .

[58]  George D. Kuh Assessing What Really Matters to Student Learning Inside The National Survey of Student Engagement , 2001 .

[59]  George Siemens,et al.  Learning Analytics , 2013 .

[60]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[61]  William J. Hussar,et al.  The Condition of Education 2012. NCES 2012-045. , 2012 .

[62]  Marie Bienkowski,et al.  Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief , 2012 .

[63]  Wendy McColskey,et al.  Measuring Student Engagement in Upper Elementary through High School: A Description of 21 Instruments. Summary. Issues & Answers. REL 2011-No. 098. , 2011 .

[64]  Shane Dawson,et al.  Mining LMS data to develop an "early warning system" for educators: A proof of concept , 2010, Comput. Educ..

[65]  R. Baker,et al.  Operationalizing and Detecting Disengagement Within Online Science Microworlds , 2015 .

[66]  Joseph B. Berger,et al.  THE ROLE OF STUDENT INVOLVEMENT AND PERCEPTIONS OF INTEGRATION IN A CAUSAL MODEL OF STUDENT PERSISTENCE , 1999 .

[67]  William J. Hussar,et al.  The Condition of Education , 2010 .

[68]  Sanghoon Park,et al.  Examining Learning Experience in Two Online Courses Using Web Logs and Experience Sampling Method (ESM) , 2015 .

[69]  Kerry Rice,et al.  A Comprehensive Look at Distance Education in the K–12 Context , 2006 .

[70]  David J. Shernoff,et al.  Further Evidence of an Engagement–Achievement Paradox Among U.S. High School Students , 2008 .

[71]  G. Ladd,et al.  Continuity and Change in Early School Engagement: Predictive of Children's Achievement Trajectories from First to Eighth Grade? , 2009, Journal of educational psychology.

[72]  Michael K. Barbour,et al.  Academic performance, course completion rates, and student perception of the quality and frequency of interaction in a virtual high school , 2013 .

[73]  Robert M. Gonyea,et al.  Unmasking the Effects of Student Engagement on First-Year College Grades and Persistence , 2008 .

[74]  David Jones,et al.  Indicators of engagement , 2010 .

[75]  James J. Appleton,et al.  Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument , 2006 .