Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem

Student engagement during classes includes behavioural, cognitive and emotional components, and is a pre-requisite for successful active learning environments. A novel approach to measuring student engagement was developed, involving triangulation of real-time student-self report, observation by trained observers and heart rate measurement. The self-report instrument was evaluated in four separate cohorts (n = 123) at Monash University and the University of North Carolina. The six item self-report demonstrated good reliability (Cronbach’s alpha values ranged from 0.7–0.81). The self-report showed predictive validity in that small group activities were rated as significantly more engaging than didactic lecturing. Additionally, there was significant inter-instructor variability and within-class variability, indicating good discrimination between classroom activities. This self-report may prove useful to academic teaching staff in evaluating and refining their active learning activities. Independent observation was not found to correlate with student self-report, due in part to students who were pretending to engage being rated as engaged by an observer. Strikingly, students reported that they were pretending to engage for 23% of class time, even for highly regarded instructors. Individual participants were rated as engaged for 42 of the 46 intervals for which they reported that they had “pretended to engage”, indicating that the two observers were unable to detect disengagement during periods in which students pretended to engage. Instructors should be aware that student cues such as eye contact and nodding may indicate pretending to engage. One particular self-report item; “I tried a new approach or way of thinking about the content”, correlated positively with heart rates, and a controlled study reproduced this finding during two activities that required students to try a new approach to understanding a concept. Agreement with this item also correlated with superior performance on two in-class written assessment tasks (n = 101, p<0.01). Further use of this tool and related educational research may be useful to identify in-class activities that are engaging and likely to lead to improved student attainment of learning outcomes.

[1]  Kristin E. Voelkl.  School Warmth, Student Participation, and Achievement , 1995 .

[2]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[3]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[4]  James J. Appleton,et al.  Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument , 2006 .

[5]  R. Pekrun.  The Control-Value Theory of Achievement Emotions: Assumptions, Corollaries, and Implications for Educational Research and Practice , 2006 .

[6]  Jan Keith Farmer,et al.  Where ' s the evidence that active learning works ? , 2006 .

[7]  Lucy Robertson.  International handbook of emotions in education , 2015 .

[8]  Benjamin C. Heddy,et al.  The Challenges of Defining and Measuring Student Engagement in Science , 2015 .

[9]  T. Kindermann,et al.  A Motivational Perspective on Engagement and Disaffection , 2009 .

[10]  George D. Kuh.  What We're Learning About Student Engagement From NSSE: Benchmarks for Effective Educational Practices , 2003 .

[11]  P. Peterson,et al.  Students’ Cognitions and Time on Task During Mathematics Instruction , 1984 .

[12]  Reinhard Pekrun,et al.  Control-Value Theory of Achievement Emotions , 2014 .

[13]  J. Connell,et al.  Relationships matter: linking teacher support to student engagement and achievement. , 2004, The Journal of school health.

[14]  Stephen W Porges,et al.  Does motor activity during psychophysiological paradigms confound the quantification and interpretation of heart rate and heart rate variability measures in young children? , 2007, Developmental psychobiology.

[15]  J. Finn.  Withdrawing From School , 1989 .

[16]  M. Chi,et al.  The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes , 2014 .

[17]  Kimberly J O'Malley,et al.  Validation Of An Observation Instrument For Measuring Student Engagement In Health Professions Settings , 2003, Evaluation & the health professions.

[18]  S. Porges.  The polyvagal theory: phylogenetic substrates of a social nervous system. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[19]  Gail R. Casper,et al.  Focus on Psychometrics Single-Item Indicators in Nursing Research , 1993 .

[20]  David J. Shernoff,et al.  Further Evidence of an Engagement–Achievement Paradox Among U.S. High School Students , 2008 .

[21]  C. Greenwood,et al.  The use of ecobehavioral assessment to identify naturally occurring effective procedures in classrooms serving students with autism and other developmental disabilities , 1991 .

[22]  Warren W. Willingham,et al.  Grades and Test Scores: Accounting for Observed Differences , 2002 .

[23]  T. Kindermann,et al.  Engagement and Disaffection in the Classroom: Part of a Larger Motivational Dynamic? , 2008 .

[24]  K. Pierce.  Posing, Pretending, Waiting for the Bell: Life in High School Classrooms , 2005 .

[25]  M. Houser,et al.  The Role of Oral Participation in Student Engagement , 2016 .

[26]  Valerie E. Lee,et al.  Effects of School Restructuring on the Achievement and Engagement of Middle-Grade Students. , 1993 .

[27]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[28]  David J. Shernoff,et al.  Student engagement in high school classrooms from the perspective of flow theory. , 2003 .

[29]  Michael J. Prince,et al.  Does Active Learning Work? A Review of the Research , 2004 .

[30]  Wendy McColskey,et al.  The Measurement of Student Engagement: A Comparative Analysis of Various Methods and Student Self-report Instruments , 2012 .

[31]  J. Michael.  Where's the evidence that active learning works? , 2006, Advances in physiology education.

[32]  Andrew J. Martin,et al.  The Motivation and Engagement Scale: Theoretical Framework, Psychometric Properties, and Applied Yields , 2012 .

[33]  Sara E. Harris,et al.  Research and Teaching: A New Tool for Measuring Student Behavioral Engagement in Large University Classes. , 2015 .

[34]  Anne C. Frenzel,et al.  The Control-Value Theory of Achievement Emotions: An Integrative Approach to Emotions in Education , 2007 .

[35]  Scott Freeman,et al.  Prescribed active learning increases performance in introductory biology , 2007, CBE life sciences education.

[36]  T. Goetz,et al.  Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research , 2002 .

[37]  Jacqueline E. McLaughlin,et al.  Student Engagement with a Flipped Classroom Teaching Design Affects Pharmacology Examination Performance in a Manner Dependent on Question Type , 2017, American Journal of Pharmaceutical Education.

[38]  Andrew J. Martin.  Examining a multidimensional model of student motivation and engagement using a construct validation approach. , 2007, The British journal of educational psychology.

[39]  Michelle K. Smith,et al.  Active learning increases student performance in science, engineering, and mathematics , 2014, Proceedings of the National Academy of Sciences.

[40]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[41]  Elizabeth Yuriev,et al.  Adopting an active learning approach to teaching in a research-intensive higher education context transformed staff teaching attitudes and behaviours , 2016 .

[42]  Som Naidu,et al.  Using active learning strategies to shift student attitudes and behaviours about learning and teaching in a research intensive educational context , 2015 .

[43]  David J. Shernoff,et al.  Student engagement in high school classrooms from the perspective of flow theory. , 2003 .