Automated gaze-based mind wandering detection during computerized learning in classrooms

AbstractWe investigate the use of commercial off-the-shelf (COTS) eye-trackers to automatically detect mind wandering—a phenomenon involving a shift in attention from task-related to task-unrelated thoughts—during computerized learning. Study 1 (N = 135 high-school students) tested the feasibility of COTS eye tracking while students learn biology with an intelligent tutoring system called GuruTutor in their classroom. We could successfully track eye gaze in 75% (both eyes tracked) and 95% (one eye tracked) of the cases for 85% of the sessions where gaze was successfully recorded. In Study 2, we used this data to build automated student-independent detectors of mind wandering, obtaining accuracies (mind wandering F1 = 0.59) substantially better than chance (F1 = 0.24). Study 3 investigated context-generalizability of mind wandering detectors, finding that models trained on data collected in a controlled laboratory more successfully generalized to the classroom than the reverse. Study 4 investigated gaze- and video- based mind wandering detection, finding that gaze-based detection was superior and multimodal detection yielded an improvement in limited circumstances. We tested live mind wandering detection on a new sample of 39 students in Study 5 and found that detection accuracy (mind wandering F1 = 0.40) was considerably above chance (F1 = 0.24), albeit lower than offline detection accuracy from Study 1 (F1 = 0.59), a finding attributable to handling of missing data. We discuss our next steps towards developing gaze-based attention-aware learning technologies to increase engagement and learning by combating mind wandering in classroom contexts.

[1]  Erik D. Reichle,et al.  Meta-awareness, perceptual decoupling and the wandering mind , 2011, Trends in Cognitive Sciences.

[2]  Evan F. Risko,et al.  Attention in Educational Contexts: The Role of the Learning Task in Guiding Attention , 2015 .

[3]  Sidney K. D'Mello,et al.  Monitoring Affect States During Effortful Problem Solving Activities , 2010, Int. J. Artif. Intell. Educ..

[4]  Sidney K. D'Mello,et al.  Giving Eyesight to the Blind: Towards Attention-Aware AIED , 2016, International Journal of Artificial Intelligence in Education.

[5]  Jianhua Lin,et al.  Divergence measures based on the Shannon entropy , 1991, IEEE Trans. Inf. Theory.

[6]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[7]  David J. Shernoff,et al.  Student engagement in high school classrooms from the perspective of flow theory. , 2003 .

[8]  Sidney K. D'Mello,et al.  What do we Think About When we Learn? , 2018, Deep Comprehension.

[9]  I. Robertson,et al.  `Oops!': Performance correlates of everyday attentional failures in traumatic brain injured and normal subjects , 1997, Neuropsychologia.

[10]  Sidney K. D'Mello,et al.  Automated Physiological-Based Detection of Mind Wandering during Learning , 2014, Intelligent Tutoring Systems.

[11]  Luca Viganò,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2015, IWSEC 2015.

[12]  Ruth Strang,et al.  How Adults Read. , 1938 .

[13]  Evelyn Barron,et al.  Absorbed in Thought , 2011, Psychological science.

[14]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[15]  Angela Stewart,et al.  Generalizability of Face-Based Mind Wandering Detection Across Task Contexts , 2017, EDM.

[16]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[17]  D. Smilek,et al.  On the Necessity of Distinguishing Between Unintentional and Intentional Mind Wandering , 2016, Psychological science.

[18]  Sidney K. D'Mello,et al.  Dialogue Modes in Expert Tutoring , 2008, Intelligent Tutoring Systems.

[19]  Andrew Olney,et al.  Student Speech Act Classification Using Machine Learning , 2011, FLAIRS.

[20]  Brandon M. Turner,et al.  When the Brain Takes a Break: A Model-Based Analysis of Mind Wandering , 2014, The Journal of Neuroscience.

[21]  Wojciech Matusik,et al.  Eye Tracking for Everyone , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Richard C. Leventhal.  The Aging Consumer: What′s all the Fuss About Anyway? , 1990 .

[23]  Takashi Matsuyama,et al.  Multi-mode saliency dynamics model for analyzing gaze and attention , 2012, ETRA.

[24]  Ryan Shaun Joazeiro de Baker,et al.  Automatic Detection of Learning-Centered Affective States in the Wild , 2015, IUI.

[25]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[26]  J. Smallwood,et al.  Counting the cost of an absent mind: Mind wandering as an underrecognized influence on educational performance , 2007, Psychonomic bulletin & review.

[27]  Clarke Rl,et al.  What's all the fuss about? , 1988, Healthcare financial management : journal of the Healthcare Financial Management Association.

[28]  Andrew Olney,et al.  Mind Wandering During Learning with an Intelligent Tutoring System , 2015, AIED.

[29]  Joanna Drummond,et al.  In the Zone: Towards Detecting Student Zoning Out Using Supervised Machine Learning , 2010, Intelligent Tutoring Systems.

[30]  Andrew Olney,et al.  Collaborative Lecturing by Human and Computer Tutors , 2010, Intelligent Tutoring Systems.

[31]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[32]  Arthur C. Graesser,et al.  Design Recommendations for Intelligent Tutoring Systems: Volume 4-Domain Modeling , 2016 .

[33]  Alan Kingstone,et al.  Everyday attention: Mind wandering and computer use during lectures , 2013, Comput. Educ..

[34]  Karl K. Szpunar,et al.  Mind wandering and education: from the classroom to online learning , 2013, Front. Psychol..

[35]  Jonathan Smallwood,et al.  The Decoupled Mind: Mind-wandering Disrupts Cortical Phase-locking to Perceptual Events , 2014, Journal of Cognitive Neuroscience.

[36]  Matthias Mittner,et al.  Linking brain networks and behavioral variability to different types of mind-wandering , 2017, Proceedings of the National Academy of Sciences.

[37]  Angela Stewart,et al.  Gaze-based Detection of Mind Wandering during Lecture Viewing , 2017, EDM.

[38]  Steve Joordens,et al.  The eyes know what you are thinking: Eye movements as an objective measure of mind wandering , 2011, Consciousness and Cognition.

[39]  Veronica Sundstedt,et al.  Simplifying game mechanics: gaze as an implicit interaction method , 2017, SIGGRAPH Asia Technical Briefs.

[40]  Benjamin W. Mooneyham,et al.  The costs and benefits of mind-wandering: a review. , 2013, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[41]  D. Schacter,et al.  Interpolated memory tests reduce mind wandering and improve learning of online lectures , 2013, Proceedings of the National Academy of Sciences.

[42]  Erik D. Reichle,et al.  Zoning Out while Reading: Evidence for Dissociations between Experience and Metaconsciousness. , 2004 .

[43]  Sidney K. D'Mello,et al.  Automatic gaze-based user-independent detection of mind wandering during computerized reading , 2016, User Modeling and User-Adapted Interaction.

[44]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[45]  E. B. Huey,et al.  Preliminary experiments in the physiology and psychology of reading , 1898 .

[46]  R. Dodge.  Visual perception during eye movement , 1900 .

[47]  D. Messinger,et al.  All smiles are positive, but some smiles are more positive than others. , 2001, Developmental psychology.

[48]  Arthur C. Graesser,et al.  Tutorial Dialog in Natural Language , 2010, Advances in Intelligent Tutoring Systems.

[49]  D. E. Irwin,et al.  Cognitive Suppression During Saccadic Eye Movements , 1996 .

[50]  Sidney K. D'Mello,et al.  Multimodal-multisensor affect detection , 2018, The Handbook of Multimodal-Multisensor Interfaces, Volume 2.

[51]  B. L. Zuber,et al.  Saccadic suppression: elevation of visual threshold associated with saccadic eye movements. , 1966, Experimental neurology.

[52]  Sidney K. D'Mello,et al.  Automatic Gaze-Based Detection of Mind Wandering during Film Viewing , 2016, EDM.

[53]  Tomasz D. Loboda,et al.  Study and Detection of Mindless Reading , 2014 .

[54]  S. D’Mello,et al.  How the stimulus influences mind wandering in semantically rich task contexts , 2018, Cognitive Research: Principles and Implications.

[55]  J. Smallwood,et al.  When attention matters: The curious incident of the wandering mind , 2008, Memory & cognition.

[56]  Arthur C. Graesser,et al.  Guru: A Computer Tutor That Models Expert Human Tutors , 2012, ITS.

[57]  E. Matin.  Saccadic suppression: a review and an analysis. , 1974, Psychological bulletin.

[58]  Manfred Tscheligi,et al.  Exploring Gaze and Hand Gestures for Non-Verbal In-Game Communication , 2017, CHI PLAY.

[59]  Jonathan Smallwood,et al.  Catching the mind in flight: Using behavioral indices to detect mindless reading in real time , 2011, Psychonomic bulletin & review.

[60]  Sidney K. D'Mello,et al.  Toward Fully Automated Person-Independent Detection of Mind Wandering , 2014, UMAP.

[61]  Nadir Weibel,et al.  Let's look at the cockpit: exploring mobile eye-tracking for observational research on the flight deck , 2012, ETRA.

[62]  M. Tinker.  How People Look at Pictures. , 1936 .

[63]  S. D’Mello,et al.  An automated behavioral measure of mind wandering during computerized reading , 2017, Behavior Research Methods.

[64]  Peter Brusilovsky.  Domain Modeling for Personalized Guidance , 2016 .

[65]  M. Kane,et al.  Tracking the train of thought from the laboratory into everyday life: An experience-sampling study of mind wandering across controlled and ecological contexts , 2009, Psychonomic bulletin & review.

[66]  T J Gawne,et al.  Activity of primate V1 cortical neurons during blinks. , 2000, Journal of neurophysiology.

[67]  Jörg Müller,et al.  Eye tracking for public displays in the wild , 2015, Personal and Ubiquitous Computing.

[68]  Kevin Fitzpatrick,et al.  Slow Fluctuations in Attentional Control of Sensory Cortex , 2011, Journal of Cognitive Neuroscience.

[69]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[70]  Jonathan Smallwood,et al.  Window to the Wandering Mind: Pupillometry of Spontaneous Thought While Reading , 2013, Quarterly journal of experimental psychology.

[71]  A W Tony Bates,et al.  Technology, e-learning, and distance education , 1995 .

[72]  Sidney K. D'Mello,et al.  "Out of the Fr-Eye-ing Pan": Towards Gaze-Based Models of Attention during Learning with Technology in the Classroom , 2017, UMAP.

[73]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[74]  J. Hoffman,et al.  The role of visual attention in saccadic eye movements , 1995, Perception & psychophysics.

[75]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[76]  Angela Stewart,et al.  Face Forward: Detecting Mind Wandering from Video During Narrative Film Comprehension , 2017, AIED.

[77]  Andrew Olney,et al.  Interactive Concept Maps and Learning Outcomes in Guru , 2012, FLAIRS.

[78]  Jingtao Wang,et al.  AttentiveLearner: Improving Mobile MOOC Learning via Implicit Heart Rate Tracking , 2015, AIED.

[79]  Frederick L Oswald,et al.  Mind-wandering, cognition, and performance: a theory-driven meta-analysis of attention regulation. , 2014, Psychological bulletin.

[80]  H. Ellis,et al.  Irrelevant thoughts, emotional mood states, and cognitive task performance , 1991, Memory & cognition.

[81]  Arthur M. Jacobs,et al.  OGAMA (Open Gaze and Mouse Analyzer): Open-source software designed to analyze eye and mouse movements in slideshow study designs , 2008, Behavior research methods.

[82]  Sidney K. D'Mello,et al.  Toward a Real-time (Day) Dreamcatcher: Detecting Mind Wandering Episodes During Online Reading , 2015, EDM.

[83]  I. E. Allen,et al.  Online Report Card: Tracking Online Education in the United States. , 2016 .

[84]  Jonathan S. A. Carriere,et al.  Out of Mind, Out of Sight , 2010, Psychological science.

[85]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[86]  Nicola C. Anderson,et al.  Everyday Attention: Variation in Mind Wandering and Memory in a Lecture , 2012 .

[87]  Diane J. Litman,et al.  When Does Disengagement Correlate with Learning in Spoken Dialog Computer Tutoring? , 2011, AIED.

[88]  Jürgen Beyerer,et al.  Gaze-based moving target acquisition in real-time full motion video , 2016, ETRA.

[89]  James R. Brockmole,et al.  Gaze-Based Signatures of Mind Wandering During Real-World Scene Processing , 2018, Journal of experimental psychology. General.

[90]  Masaru Ishii,et al.  Surgical Task and Skill Classification from Eye Tracking and Tool Motion in Minimally Invasive Surgery , 2010, MICCAI.

[91]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[92]  Sidney K. D'Mello,et al.  Automatic Detection of Mind Wandering from Video in the Lab and in the Classroom , 2019, IEEE Transactions on Affective Computing.

[93]  Arthur C. Graesser,et al.  Mind wandering while reading easy and difficult texts , 2013, Psychonomic bulletin & review.

[94]  Oleg V. Komogortsev,et al.  Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network , 2010, CHI Extended Abstracts.

[95]  J. Smallwood,et al.  The restless mind. , 2006, Psychological bulletin.

[96]  Andrew Olney,et al.  Mining Collaborative Patterns in Tutorial Dialogues , 2010, EDM 2010.

[97]  F. Campbell,et al.  Saccadic omission: Why we do not see a grey-out during a saccadic eye movement , 1978, Vision Research.

[98]  Janet Ainley,et al.  The role of attention in expert classroom practice , 2007 .

[99]  Steve Majerus,et al.  Mind-wandering: phenomenology and function as assessed with a novel experience sampling method. , 2011, Acta psychologica.

[100]  E. Javal,et al.  Essai sur la physiologie de la lecture , 1878 .

[101]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[102]  Caitlin Mills,et al.  Neural Dynamics of Spontaneous Thought: An Electroencephalographic Study , 2017, HCI.

[103]  Sidney K. D'Mello,et al.  The Eyes Have It: Gaze-based Detection of Mind Wandering during Learning with an Intelligent Tutoring System , 2016, EDM.

[104]  L. Giambra,et al.  A Laboratory Method for Investigating Influences on Switching Attention to Task-Unrelated Imagery and Thought , 1995, Consciousness and Cognition.

[105]  Jonathan Smallwood,et al.  Going AWOL in the Brain: Mind Wandering Reduces Cortical Analysis of External Events , 2008, Journal of Cognitive Neuroscience.

[106]  James Hays,et al.  WebGazer: Scalable Webcam Eye Tracking Using User Interactions , 2016, IJCAI.

[107]  M. Kane,et al.  Does mind wandering reflect executive function or executive failure? Comment on Smallwood and Schooler (2006) and Watkins (2008). , 2010, Psychological bulletin.

[108]  Lisa Miller,et al.  Mindfulness Interventions with Youth: A Meta-Analysis , 2015 .

[109]  M. Kane,et al.  Conducting the train of thought: working memory capacity, goal neglect, and mind wandering in an executive-control task. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[110]  Erik D. Reichle,et al.  Eye Movements During Mindless Reading , 2010, Psychological science.

[111]  E. B. Huey,et al.  The Psychology and Pedagogy of Reading, with a Review of the History of Reading and Writing and of Methods, Texts, and Hygiene in Reading , 2007 .

[112]  H. Deubel,et al.  Saccade target selection and object recognition: Evidence for a common attentional mechanism , 1996, Vision Research.

[113]  Michael J Kane,et al.  Drifting from slow to "D'oh!": working memory capacity and mind wandering predict extreme reaction times and executive control errors. , 2012, Journal of experimental psychology. Learning, memory, and cognition.

[114]  J. Smallwood,et al.  The science of mind wandering: empirically navigating the stream of consciousness. , 2015, Annual review of psychology.