Using eye movements to model the sequence of text-picture processing for multimedia comprehension

This study used eye movement modeling examples (EMME) to support students’ integrative processing of verbal and graphical information during the reading of an illustrated text. EMME consists of a replay of eye movements of a model superimposed onto the materials that are processed for accomplishing the task. Specifically, the study investigated the effects of modeling the temporal sequence of text and picture processing as shown in various replays of a model’s gazes. Eighty-four 7th graders were randomly assigned to one of the four experimental conditions: text-first processing sequence (text-first EMME), picture-first processing sequence (picture-first EMME), picture-last processing sequence (picture-last EMME) and no-EMME (control). Online and offline measures were used. Eye movement indices indicate that only readers in the picture-first EMME condition spent significantly longer processing the picture and showed stronger integrative processing of verbal and graphical information than students in the no-EMME condition. Moreover, readers in all EMME conditions outperformed those in the control condition for recall. However, for learning and transfer, only readers in the picture-first EMME condition were significantly superior to readers of the control condition. Furthermore, both the frequency and duration of integrative processing of verbal and graphical information mediated the effect of condition on learning outcomes.

[1]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[2]  Ralph E. Reynolds,et al.  The Nature of the Refutation Text Effect: An Investigation of Attention Allocation , 2010 .

[3]  Patrik Pluchino,et al.  Eye-movement modeling of integrative reading of an illustrated text: Effects on processing and learning , 2015 .

[4]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.

[5]  Patrik Pluchino,et al.  Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns , 2013, Comput. Educ..

[6]  Bradley W. Bergey,et al.  Effects of three diagram instruction methods on transfer of diagram comprehension skills: The critical role of inference while learning , 2013 .

[7]  Rolf Ploetzner,et al.  Supporting learning from illustrated texts: conceptualizing and evaluating a learning strategy , 2011 .

[8]  Richard E. Mayer,et al.  Cognitive Theory of Multimedia Learning , 2021, The Cambridge Handbook of Multimedia Learning.

[9]  Patrik Pluchino,et al.  Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: an eye-movement study , 2015 .

[10]  Katharina Scheiter,et al.  How Inspecting a Picture Affects Processing of Text in Multimedia Learning , 2013 .

[11]  Rolf Ploetzner,et al.  What contributes to the split-attention effect? The role of text segmentation, picture labelling, and spatial proximity , 2010 .

[12]  T. Gog,et al.  Learning to see: Guiding students' attention via a Model's eye movements fosters learning , 2013 .

[13]  K. Rayner.  The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[14]  Damien Litchfield,et al.  Rapid communication: Using another's gaze as an explicit aid to insight problem solving , 2011, Quarterly journal of experimental psychology.

[15]  Marcus Nyström,et al.  How a picture facilitates the process of learning from text: Evidence for scaffolding , 2013 .

[16]  Detlev Leutner,et al.  The role of spatial ability in learning from instructional animations - Evidence for an ability-as-compensator hypothesis , 2011, Comput. Hum. Behav..

[17]  Patrik Pluchino,et al.  Effects of Picture Labeling on Science Text Processing and Learning: Evidence From Eye Movements , 2013 .

[18]  J. Hyönä,et al.  Utilization of Illustrations during Learning of Science Textbook Passages among Low- and High-Ability Children. , 1999, Contemporary educational psychology.

[19]  W. Schnotz,et al.  Strategy Shifts during Learning from Texts and Pictures. , 2014 .

[20]  Erol Özçelik,et al.  An eye-tracking study of how color coding affects multimedia learning , 2009, Comput. Educ..

[21]  Jürgen Baumert,et al.  Teachers' Beliefs, Instructional Behaviors, and Students' Engagement in Learning from Texts with Instructional Pictures. , 2011 .

[22]  Kirsten R. Butcher.  The Cambridge Handbook of Multimedia Learning: The Multimedia Principle , 2014 .

[23]  Rainer Bromme,et al.  Coherence Formation when Learning from Text and Pictures: What Kind of Support for Whom? , 2009 .

[24]  Silke Schworm,et al.  Learning with video-based examples - Are you sure you do not need help? , 2014, J. Comput. Assist. Learn..

[25]  J. D. Fletcher,et al.  The Multimedia Principle. , 2005 .

[26]  Mayu Nishimura,et al.  What are you looking at? Acuity for triadic eye gaze. , 2004, The Journal of general psychology.

[27]  Jennifer Grace Cromley,et al.  Reading Achievement and Science Proficiency: International Comparisons From the Programme on International Student Assessment , 2009 .

[28]  Patrik Pluchino,et al.  Using eye-tracking technology as an indirect instruction tool to improve text and picture processing and learning , 2016, Br. J. Educ. Technol..

[29]  Panayiota Kendeou,et al.  Augmenting the Refutation Text Effect with Analogies and Graphics , 2016 .

[30]  Wolfgang Schnotz,et al.  Integrated model of text and picture comprehension 1 Running head: INTEGRATED MODEL OF TEXT AND PICTURE COMPREHENSION Chapter 4 Integrated Model of Text and Picture Comprehension , 2013 .

[31]  T. Crawford,et al.  Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection. , 2010, Journal of experimental psychology. Applied.

[32]  Katharina Scheiter,et al.  Attention guidance during example study via the model's eye movements , 2009, Comput. Hum. Behav..

[33]  Andy Field,et al.  Discovering statistics using SPSS, 2nd ed. , 2005 .

[34]  K. Scheiter,et al.  Conveying clinical reasoning based on visual observation via eye-movement modelling examples , 2012, Instructional Science.

[35]  Russell N. Carney,et al.  Pictorial Illustrations Still Improve Students' Learning from Text , 2002 .

[36]  Jukka Hyönä,et al.  Effects of Topic Headings on Text Processing: Evidence from Adult Readers' Eye Fixation Patterns. , 2004 .

[37]  Andreas Gegenfurtner,et al.  Seeing through a teacher’s eyes improves students’ imaging interpretation , 2012, Medical education.

[38]  T. Gog,et al.  In the eyes of the beholder: How experts and novices interpret dynamic stimuli , 2010 .

[39]  Katharina Scheiter,et al.  Picture or Text First? Explaining Sequence Effects when Learning with Pictures and Text , 2015 .

[40]  Jennifer Grace Cromley,et al.  Cognitive activities in complex science text and diagrams , 2010 .

[41]  Cheryl I. Johnson,et al.  An eye movement analysis of the spatial contiguity effect in multimedia learning. , 2012, Journal of experimental psychology. Applied.

[42]  A. Meltzoff,et al.  The development of gaze following and its relation to language. , 2005, Developmental science.

[43]  S. Ainsworth.  DeFT: A Conceptual Framework for Considering Learning with Multiple Representations. , 2006 .

[44]  Jan-Willem Strijbos,et al.  Inferring mindful cognitive-processing of peer-feedback via eye-tracking: role of feedback-characteristics, fixation-durations and transitions , 2015, J. Comput. Assist. Learn..

[45]  K. Scheiter,et al.  Signals foster multimedia learning by supporting integration of highlighted text and diagram elements , 2015 .

[46]  Gregory Schraw,et al.  The use of adjunct displays to facilitate comprehension of causal relationships in expository text , 2009 .

[47]  S. Vandenberg,et al.  Mental Rotations, a Group Test of Three-Dimensional Spatial Visualization , 1978, Perceptual and motor skills.

[48]  M. Bannert,et al.  Construction and interference in learning from multiple representation , 2003 .

[49]  W. Schnotz.  Integrated Model of Text and Picture Comprehension , 2021, The Cambridge Handbook of Multimedia Learning.