Scientific Visualizations Based on Integrated Model of Text and Picture Comprehension via Eye-tracking

Abstract This study investigates the “color combination effect” and the “sequence of pictures and text effect” in the “Integrated Model of Text and Picture Comprehension” (ITPC) on the formation of the mental model and cognitive load in both desktop and mobile devices. These effects are studied using an experiment including scientific contexts with specific color combinations and different picture and text sequences. The participants of this experiment are graduate students from different fields of science in Universiti Sains Malaysia. Four tasks are assigned for each participant in each phase and every task includes a scientific context accompanied with visualization. After each task the participant is asked to answer a set of questions related to the context. Eye-tracking methods are adopted to record gaze data of the participants while reading each context. Analysis of the gaze data and the percentage of the correct answers to the questions indicate that the sequence of picture and text and use of color in the visualizations results in reduction of the cognitive load of the participants.

[1]  Chaomei Chen,et al.  Visualizing knowledge domains , 2005, Annu. Rev. Inf. Sci. Technol..

[2]  Edward J. Kameenui,et al.  Effects of Teacher-Constructed Pre- and Post-Graphic Organizer Instruction on Sixth-Grade Science Students’ Comprehension and Recall , 1988 .

[3]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[4]  Michael Mayrath,et al.  Graphic organizers or graphic overviews? Presentation order effects with computer-based text , 2012 .

[5]  Robert T. Solman,et al.  The Influence of Illustrations on Children's Comprehension of Written Stories , 1987 .

[6]  Mark Ullrich.  Einflüsse der Verarbeitungsreihenfolge auf den Wissenserwerb mit Texten und Bildern , 2011 .

[7]  Mary Hegarty,et al.  Effects of knowledge and display design on comprehension of complex graphics , 2010 .

[8]  Raymond S. Dean,et al.  Pictorial organization in prose learning , 1983 .

[9]  Thierry Baccino,et al.  Do graphical overviews facilitate or hinder comprehension in hypertext? , 2009, Comput. Educ..

[10]  Marcus Nyström,et al.  How a picture facilitates the process of learning from text: Evidence for scaffolding , 2013 .

[11]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[12]  Wolfgang Schnotz,et al.  External and internal representations in the acquisition and use of knowledge: visualization effects on mental model construction , 2008 .

[13]  J. D. Fletcher,et al.  The Multimedia Principle. , 2005 .

[14]  Marcia K. Johnson,et al.  Contextual prerequisites for understanding: Some investigations of comprehension and recall , 1972 .

[15]  M. A. Borges,et al.  Contextual and Motivational CUE Effects on the Comprehension and Recall of Prose , 1980 .

[16]  Gregory Schraw,et al.  The Effect of Diagrams on Online Reading Processes and Memory , 2011 .

[17]  Heinrich Hußmann,et al.  Mobile or desktop websites?: website usage on multitouch devices , 2010, NordiCHI.

[18]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[19]  Daniel H. Robinson,et al.  Visual argument: Graphic organizers are superior to outlines in improving learning from text. , 1995 .

[20]  Andrew D. Katayama,et al.  Interactive Effects of Graphic Organizers and Delayed Review on Concept Application , 1998 .

[21]  Stephanie Boehm,et al.  Information Dashboard Design The Effective Visual Communication Of Data , 2016 .

[22]  Gregory Schraw,et al.  The use of adjunct displays to facilitate comprehension of causal relationships in expository text , 2009 .

[23]  Pat Hanrahan,et al.  Measuring the task-evoked pupillary response with a remote eye tracker , 2008, ETRA.