Adaptive multimedia: Using gaze-contingent instructional guidance to provide personalized processing support

Abstract The goal of the study was to develop an adaptive, gaze-contingent learning environment that would support learners in their information-processing behavior when learning from illustrated texts. To this end, three experiments were conducted. In Experiment 1 (N = 32) three groups of learners were identified based on differences in their eye movements obtained while they were learning with a non-adaptive multimedia learning environment. The group of learners who displayed longer fixations times and higher fixations counts on text and pictures as well as more text-picture transitions had better learning outcomes than a group with a less intense information-processing behavior. These findings were used to develop a gaze-contingent adaptive system. It analyzes learners’ eye movements during learning in real time and - in case of poor information processing (i.e., behavior similar to that of the unsuccessful learner group in Experiment 1) - alters the presentation of the materials in a way that is expected to trigger a more adequate processing (e.g., by highlighting relations between text and pictures). In Experiment 2 (N = 79) and Experiment 3 (N = 62) the adaptive multimedia learning system was compared to a non-adaptive, static presentation of the same materials. Experiment 2 showed no differences between both learning systems in terms of learning outcome. In Experiment 3, where the thresholds for adaptive responses were slightly modified, the gaze-based adaptive system hindered learners with weaker cognitive prerequisites, but tended to support learners with stronger cognitive prerequisites. Possible reasons are discussed and future research directions suggested.

[1]  Julie Thomas,et al.  Attention aware systems: Theories, applications, and research agenda , 2006, Comput. Hum. Behav..

[2]  Alexander Renkl,et al.  How to Design Adaptive Information Environments to Support Self-Regulated Learning with Multimedia , 2017 .

[3]  Katharina Scheiter,et al.  Self‐regulated learning from illustrated text: Eye movement modelling to support use and regulation of cognitive processes during learning from multimedia , 2018, The British journal of educational psychology.

[4]  R. Bjork,et al.  Self-regulated learning: beliefs, techniques, and illusions. , 2013, Annual review of psychology.

[5]  Erol Özçelik,et al.  Why does signaling enhance multimedia learning? Evidence from eye movements , 2010, Comput. Hum. Behav..

[6]  Katharina Scheiter,et al.  Signaling Text-Picture Relations in Multimedia Learning: The Influence of Prior Knowledge. , 2017 .

[7]  Ok-Choon Park,et al.  Adaptive Instructional Systems , 2007 .

[8]  Heiko Rölke,et al.  The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. , 2014 .

[9]  Patrik Pluchino,et al.  Eye-movement modeling of integrative reading of an illustrated text: Effects on processing and learning , 2015 .

[10]  Alexander Toet,et al.  Gaze directed displays as an enabling technology for attention aware systems , 2006, Comput. Hum. Behav..

[11]  R. Mayer,et al.  For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. , 1994 .

[12]  A. Renkl.  Worked-out examples: instructional explanations support learning by self- explanations , 2002 .

[13]  Alexander Eitel,et al.  How repeated studying and testing affects multimedia learning: Evidence for adaptation to task demands , 2016 .

[14]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[15]  A. Corbett,et al.  The Cambridge Handbook of the Learning Sciences: Cognitive Tutors , 2005 .

[16]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[17]  T. O. Nelson.  Consciousness and metacognition. , 1996 .

[18]  Geoffrey I. Webb,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Machine Learning for User Modeling , 1999 .

[19]  K. Koedinger,et al.  Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system , 2011, Learning and Instruction.

[20]  Richard E. Mayer,et al.  Multimedia Learning , 2001, Visible Learning Guide to Student Achievement.

[21]  Katharina Scheiter,et al.  Studying Visual Displays: How to Instructionally Support Learning , 2017 .

[22]  P. Spargo,et al.  Construction of a paper-and-pencil Test of Basic Scientific Literacy based on selected literacy goals recommended by the American Association for the Advancement of Science , 1996 .

[23]  Katharina Scheiter,et al.  The Role of Working Memory in Multimedia Instruction: Is Working Memory Working During Learning from Text and Pictures? , 2011 .

[24]  Andrew Olney,et al.  Gaze tutor: A gaze-reactive intelligent tutoring system , 2012, Int. J. Hum. Comput. Stud..

[25]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[26]  Katharina Scheiter,et al.  Signaling text-picture relations in multimedia learning: A comprehensive meta-analysis , 2016 .

[27]  Arthur C. Graesser,et al.  AutoTutor: an intelligent tutoring system with mixed-initiative dialogue , 2005, IEEE Transactions on Education.

[28]  K. Scheiter,et al.  Signals foster multimedia learning by supporting integration of highlighted text and diagram elements , 2015 .

[29]  S. C. Johnson.  Hierarchical clustering schemes , 1967, Psychometrika.

[30]  Yavuz Akbulut,et al.  Adaptive educational hypermedia accommodating learning styles: A content analysis of publications from 2000 to 2011 , 2012, Comput. Educ..

[31]  Patrik Pluchino,et al.  Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns , 2013, Comput. Educ..

[32]  Albert T. Corbett,et al.  A Cognitive Tutor for Genetics Problem Solving: Learning Gains and Student Modeling , 2010 .

[33]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[34]  Anthony Jameson,et al.  Numerical uncertainty management in user and student modeling: An overview of systems and issues , 2005, User Modeling and User-Adapted Interaction.

[35]  Katharina Scheiter,et al.  Does a Strategy Training Foster Students’ Ability to Learn From Multimedia? , 2015 .

[36]  Katharina Scheiter,et al.  The Use of Eye Tracking as a Research and Instructional Tool in Multimedia Learning , 2017 .

[37]  Vincent Aleven,et al.  Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor , 2006, Int. J. Artif. Intell. Educ..

[38]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[39]  Vincent Aleven,et al.  Instruction Based on Adaptive Learning Technologies , 2016 .

[40]  J. Hyönä,et al.  Utilization of Illustrations during Learning of Science Textbook Passages among Low- and High-Ability Children. , 1999, Contemporary educational psychology.

[41]  Peter Gerjets,et al.  Extending multimedia research: How do prerequisite knowledge and reading comprehension affect learning from text and pictures , 2014, Comput. Hum. Behav..

[42]  Paul Ginns.  Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects , 2006 .

[43]  Eric Jamet,et al.  An eye-tracking study of cueing effects in multimedia learning , 2014, Comput. Hum. Behav..

[44]  Paul Chandler,et al.  Levels of Expertise and Instructional Design , 1998, Hum. Factors.

[45]  R. Mayer,et al.  Interactive Multimodal Learning Environments , 2007 .

[46]  A. Renkl,et al.  How multiple external representations are used and how they can be made more useful , 2009 .

[47]  Slava Kalyuga,et al.  The Expertise Reversal Effect , 2003 .

[48]  Tamara van Gog,et al.  Improving self-monitoring and self-regulation: From cognitive psychology to the classroom , 2012 .

[49]  Nicholas V. Mudrick,et al.  Using Data Visualizations to Foster Emotion Regulation During Self-Regulated Learning with Advanced Learning Technologies , 2017 .

[50]  S. West,et al.  Multiple Regression: Testing and Interpreting Interactions. , 1994 .

[51]  Yeung,et al.  Cognitive Load and Learner Expertise: Split-Attention and Redundancy Effects in Reading with Explanatory Notes , 1998, Contemporary educational psychology.

[52]  Patrik Pluchino,et al.  Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study , 2015 .

[53]  V. Shute,et al.  Adaptive Technologies for Training and Education: Adaptive Educational Systems , 2012 .

[54]  Jan L. Plass,et al.  Learning from multiple representations: An examination of fixation patterns in a science simulation , 2014, Comput. Hum. Behav..

[55]  Jennifer Grace Cromley,et al.  Cognitive activities in complex science text and diagrams , 2010 .

[56]  Cheryl I. Johnson,et al.  An eye movement analysis of the spatial contiguity effect in multimedia learning. , 2012, Journal of experimental psychology. Applied.

[57]  Jukka Hyönä,et al.  The Use of Eye Movements in the Study of Multimedia Learning. , 2010 .

[58]  Katharina Scheiter,et al.  Implementation intentions during multimedia learning: Using if-then plans to facilitate cognitive processing , 2015 .

[59]  Cristina Conati,et al.  Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation , 2007, Knowl. Based Syst..

[60]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[61]  Holger Schmidt,et al.  An adaptive and adaptable learning platform with realtime eye-tracking support: lessons learned , 2014, DeLFI.

[62]  Katharina Scheiter,et al.  Eye tracking as a tool to study and enhance multimedia learning , 2010 .