Carry-Forward Effect: Early scaffolding learning processes

Multimodal data enables us to capture the cognitive and affective states of students to provide a holistic understanding of learning processes in a wide variety of contexts. With the use of sensing technology, we can capture learners’ states in near real-time and support learning. Moreover, multimodal data allows us to obtain early-predictions of learning performance, and support learning in a timely manner. In this contribution, we utilize the notion of “carry forward effect”, an inferential and predictive modelling approach that utilizes multimodal data measurements detrimental to learning performance to provide timely feedback suggestions. carry forward effect can provide a way to prioritize conflicting feedback suggestions in a multimodal data based scaffolding tool. We showcase the empirical proof of carry forward effect with the use of two different learning scenarios: debugging and game-based learning.

[1]  M. Giannakos,et al.  Keep Calm and Do Not Carry-Forward: Toward Sensor-Data Driven AI Agent to Enhance Human Learning , 2022, Frontiers in Artificial Intelligence.

[2]  M. Giannakos,et al.  Investigating gaze interaction to support children’s gameplay , 2021 .

[3]  M. Giannakos,et al.  Children’s Play and Problem Solving in Motion-Based Educational Games: Synergies between Human Annotations and Multi-Modal Data , 2021, IDC.

[4]  Kshitij Sharma,et al.  Children's Interaction with Motion-Based Touchless Games: Kinecting Effectiveness and Efficiency , 2020, CHI PLAY.

[5]  KostakosVassilis,et al.  Assessing Cognitive Performance Using Physiological and Facial Features , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[6]  Kshitij Sharma,et al.  Assessing Cognitive Performance Using Physiological and Facial Features , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[7]  Mohammad Saiedur Rahaman,et al.  n-Gage: Predicting in-class Emotional, Behavioural and Cognitive Engagement in the Wild. , 2020 .

[8]  Kshitij Sharma,et al.  Multimodal data capabilities for learning: What can multimodal data tell us about learning? , 2020, Br. J. Educ. Technol..

[9]  Ulrik Schroeder,et al.  Eye-Tracking in Educational Multi-Touch Games: Design-Based (interaction) research and great visions , 2020, ETRA Short Papers.

[10]  P. Dillenbourg,et al.  Eye-tracking and artificial intelligence to enhance motivation and learning , 2020, Smart Learning Environments.

[11]  Kshitij Sharma,et al.  Utilizing Interactive Surfaces to Enhance Learning, Collaboration and Engagement: Insights from Learners’ Gaze and Speech , 2020, Sensors.

[12]  Chih‐Hung Chen,et al.  AR videos as scaffolding to foster students' learning achievements and motivation in EFL learning , 2020, Br. J. Educ. Technol..

[13]  Omid Noroozi,et al.  Multimodal data to design visual learning analytics for understanding regulation of learning , 2019, Comput. Hum. Behav..

[14]  Andreas Bulling,et al.  Classifying Attention Types with Thermal Imaging and Eye Tracking , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[15]  Andreas Janson,et al.  Ladders for Learning: Is Scaffolding the Key to Teaching Problem-Solving in Technology-Mediated Learning Contexts? , 2019 .

[16]  Nitesh V. Chawla,et al.  Differentiating Higher and Lower Job Performers in the Workplace Using Mobile Sensing , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[17]  Albrecht Schmidt,et al.  Your skin resists: exploring electrodermal activity as workload indicator during manual assembly , 2019, EICS.

[18]  Nigel Bosch,et al.  Automated gaze-based mind wandering detection during computerized learning in classrooms , 2019, User Modeling and User-Adapted Interaction.

[19]  Elena Di Lascio,et al.  Using Unobtrusive Wearable Sensors to Measure the Physiological Synchrony Between Presenters and Audience Members , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[20]  Jorge Gonçalves,et al.  Measuring the Effects of Stress on Mobile Interaction , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[21]  Veljko Pejovic,et al.  My Watch Says I'm Busy: Inferring Cognitive Load with Low-Cost Wearables , 2018, UbiComp/ISWC Adjunct.

[22]  Kursat Cagiltay,et al.  A systematic review of eye tracking research on multimedia learning , 2018, Comput. Educ..

[23]  Elena Di Lascio,et al.  Unobtrusive Assessment of Students' Emotional Engagement during Lectures Using Electrodermal Activity Sensors , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[24]  Yongqiang Lyu,et al.  Touch Sense , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[25]  Suranga Nanayakkara,et al.  Going beyond performance scores: understanding cognitive-affective states in kindergarteners , 2018, IDC.

[26]  Heather Toomey Zimmerman,et al.  A Methodological Case Study with Mobile Eye-Tracking of Child Interaction in a Science Museum , 2018, TechTrends.

[27]  Sanit Haruehansawasin,et al.  Scaffolding in problem-based learning for low-achieving learners , 2018 .

[28]  Martin Raubal,et al.  The Index of Pupillary Activity: Measuring Cognitive Load vis-à-vis Task Difficulty with Pupil Oscillation , 2018, CHI.

[29]  Alexander Maedche,et al.  Got Flow?: Using Machine Learning on Physiological Data to Classify Flow , 2018, CHI Extended Abstracts.

[30]  JeongGil Ko,et al.  EngageMon: Multi-Modal Engagement Sensing for Mobile Games , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[31]  Jan Ole Johanssen,et al.  Employing Consumer Wearables to Detect Office Workers' Cognitive Load for Interruption Management , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[32]  Michail N. Giannakos,et al.  Gaze insights into debugging behavior using learner-centred analysis , 2018, LAK.

[33]  Sónia Matos,et al.  Gaming Versus Storytelling: Understanding Children's Interactive Experiences in a Museum Setting , 2017, ICIDS.

[34]  Chuen-Tsai Sun,et al.  Customizing scaffolds for game-based learning in physics: Impacts on knowledge acquisition and game design creativity , 2017, Comput. Educ..

[35]  Laura A. Dabbish,et al.  Supporting Social Interactions with an Expressive Heart Rate Sharing Application , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[36]  Susan R. Fussell,et al.  Intelligent Interruption Management using Electro Dermal Activity based Physiological Sensor for Collaborative Sensemaking , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[37]  Deepak Ganesan,et al.  iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[38]  Bertrand Schneider,et al.  Using Mobile Eye-Trackers to Unpack the Perceptual Benefits of a Tangible User Interface for Collaborative Learning , 2016, ACM Trans. Comput. Hum. Interact..

[39]  Kai Kunze,et al.  Estimation of english skill with a mobile eye tracker , 2016, UbiComp Adjunct.

[40]  Andreas Dengel,et al.  Towards an intelligent textbook: eye gaze based attention extraction on materials for learning and instruction in physics , 2016, UbiComp Adjunct.

[41]  Darren Gergle,et al.  Gazed and Confused: Understanding and Designing Shared Gaze for Remote Collaboration , 2016, CHI.

[42]  Fengfeng Ke,et al.  Designing and integrating purposeful learning in game play: a systematic review , 2015, Educational Technology Research and Development.

[43]  R. Nager,et al.  Skin temperature reveals the intensity of acute stress , 2015, Physiology & Behavior.

[44]  Marcelo Worsley,et al.  Leveraging multimodal learning analytics to differentiate student learning strategies , 2015, LAK.

[45]  Agata Rozga,et al.  Using electrodermal activity to recognize ease of engagement in children during social interactions , 2014, UbiComp.

[46]  Bryan Reimer,et al.  Classifying driver workload using physiological and driving performance data: two field studies , 2014, CHI.

[47]  René F. Kizilcec,et al.  Showing face in video instruction: effects on information retention, visual attention, and affect , 2014, CHI.

[48]  Cristina Conati,et al.  Understanding Attention to Adaptive Hints in Educational Games: An Eye-Tracking Study , 2013, International Journal of Artificial Intelligence in Education.

[49]  Slava Kalyuga,et al.  Using segmentation to support the learning from animated soccer scenes: An effect of prior knowledge , 2013 .

[50]  Dominik J. Leiner,et al.  EDA Positive Change: A Simple Algorithm for Electrodermal Activity to Measure General Audience Arousal During Media Exposure , 2012 .

[51]  Richard E. Mayer,et al.  Affective impact of navigational and signaling aids to e-learning , 2012, Comput. Hum. Behav..

[52]  Azizah Jaafar,et al.  Eye Tracking in Educational Games Environment: Evaluating User Interface Design through Eye Tracking Patterns , 2011, IVIC.

[53]  Michael J. Hannafin,et al.  Scaffolding problem solving in technology-enhanced learning environments (TELEs): Bridging research and theory with practice , 2011, Comput. Educ..

[54]  Andrea Bonarini,et al.  Enjoyment recognition from physiological data in a car racing game , 2010, AFFINE '10.

[55]  Jean-Pierre Thibaut,et al.  Cognitive load and semantic analogies: Searching semantic space , 2010, Psychonomic bulletin & review.

[56]  Jonathan I. Maletic,et al.  An Eye Tracking Study on camelCase and under_score Identifier Styles , 2010, 2010 IEEE 18th International Conference on Program Comprehension.

[57]  Jean-Pierre Thibaut,et al.  The development of analogy making in children: cognitive load and executive functions. , 2010, Journal of experimental child psychology.

[58]  M. Volman,et al.  Scaffolding in Teacher–Student Interaction: A Decade of Research , 2010 .

[59]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[60]  L. Davachi,et al.  Enhanced Intersubject Correlations during Movie Viewing Correlate with Successful Episodic Encoding , 2008, Neuron.

[61]  Michael J. Hannafin,et al.  Scaffolding in technology-enhanced learning environments , 2007, Interact. Learn. Environ..

[62]  Y. Kafai,et al.  Children’s Participation in a Virtual Epidemic in the Science Classroom: Making Connections to Natural Infectious Diseases , 2006, Journal of science education and technology.

[63]  Kursat Cagiltay,et al.  Scaffolding strategies in electronic performance support systems: types and challenges , 2006 .

[64]  F. Paas,et al.  Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. , 2005, Journal of experimental psychology. Applied.

[65]  Paul Ginns Meta-Analysis of the Modality Effect. , 2005 .

[66]  B. Velichkovsky,et al.  Time course of information processing during scene perception: The relationship between saccade amplitude and fixation duration , 2005 .

[67]  C. Hmelo‐Silver Problem-Based Learning: What and How Do Students Learn? , 2004 .

[68]  Joseph Krajcik,et al.  A Scaffolding Design Framework for Software to Support Science Inquiry , 2004, The Journal of the Learning Sciences.

[69]  Brian J. Reiser,et al.  Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work , 2004, The Journal of the Learning Sciences.

[70]  James E. Driskell,et al.  Games, Motivation, and Learning: A Research and Practice Model , 2002 .

[71]  Pablo Romero,et al.  Co-ordination of multiple external representations during Java program debugging , 2002, Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments.

[72]  Kevin Oliver,et al.  Student Management of Web-Based Hypermedia Resources During Open-Ended Problem Solving , 2000 .

[73]  M. Csíkszentmihályi Finding Flow: The Psychology of Engagement with Everyday Life , 1997 .

[74]  C. Kennard,et al.  Predictive eye saccades are different from visually triggered saccades , 1987, Vision Research.

[75]  J. Bruner,et al.  The role of tutoring in problem solving. , 1976, Journal of child psychology and psychiatry, and allied disciplines.

[76]  Antony William Joseph,et al.  Potential Eye Tracking Metrics and Indicators to Measure Cognitive Load in Human-Computer Interaction Research , 2020, Journal of scientific research.

[77]  David M. Weiss,et al.  High School Students' Collaboration and Engagement With Scaffolding and Information as Predictors of Argument Quality During Problem-Based Learning , 2017, CSCL.

[78]  Mahadev Satyanarayanan,et al.  OpenFace: A general-purpose face recognition library with mobile applications , 2016 .

[79]  Herre van Oostendorp,et al.  A meta-analytic review of the role of instructional support in game-based learning , 2013, Comput. Educ..

[80]  Patrick Jermann,et al.  Understanding Collaborative Program Comprehension: Interlacing Gaze and Dialogues , 2013, CSCL.

[81]  Patrick Jermann,et al.  Gaze Evidence for Different Activities in Program Understanding , 2012, PPIG.

[82]  Anneli Olsen The Tobii IVT Fixation Filter Algorithm description , 2012 .

[83]  Ton de Jong,et al.  Instructional support in games , 2011, CGAMES 2011.

[84]  Linden J. Ball,et al.  Eye tracking in HCI and usability research. , 2006 .

[85]  A. C. Smit,et al.  A short-latency transition in saccade dynamics during square-wave tracking and its significance for the differentiation of visually-guided and predictive saccades , 2004, Experimental Brain Research.

[86]  Noriaki Harada,et al.  Cold-stress tests involving finger skin temperature measurement for evaluation of vascular disorders in hand-arm vibration syndrome: review of the literature , 2002, International archives of occupational and environmental health.

[87]  A. Collins,et al.  COGNITIVE APPRENTICESHIP: MAKING THINKING VISIBLE , 1991 .