Colorful Bouquets in Multimedia ­Research: A Closer Look at the Modality Effect

The modality effect states that students learn better from multimedia instructional messages when the text is spoken rather than when it is written. The effect is considered as a derivate of the multimedia effect, because its rationale is to take full advantage of text-picture combinations by maximizing temporal contiguity of verbal and pictorial information in working memory and by minimizing negative effects of split attention. Accordingly, the modality effect is interrelated to other multimedia learning effects, interrelated to the degree of literacy and interrelated to research on reading comprehension and listening comprehension. The modality effect is then analy- zed in terms of sensorial and cognitive processing. The sensorial facet of split attention refers to the total amount of information transferred to working memory under severely limited time constraints. The cognitive facet of split attention refers to the simultaneous availability of verbal and pictorial information in working memory as part of a fleeting database for mental model construction. Other explanations can be derived from these facets. The modality effect turns out to be not a unitary effect based on a coherent set of causal relationships. Instead, it aggregates seemingly similar observations that are in fact based on different processing mechanisms. It seems that it can easily be suspended or changed to the opposite under specific conditions.

[1]  Wolfgang Schnotz,et al.  The Cambridge Handbook of Multimedia Learning: An Integrated Model of Text and Picture Comprehension , 2005 .

[2]  Katharina Scheiter,et al.  Boundary Conditions and Constraints of the Modality Effect , 2011 .

[3]  J. Hyönä,et al.  Utilization of Illustrations during Learning of Science Textbook Passages among Low- and High-Ability Children. , 1999, Contemporary educational psychology.

[4]  P. Chandler,et al.  Cognitive Load While Learning to Use a Computer Program , 1996 .

[5]  Thomas E. Scruggs,et al.  Constructing more meaningful relationships: Mnemonic instruction for special populations , 1989 .

[6]  P. Chandler,et al.  When auditory presentations should and should not be a component of multimedia instruction , 2003 .

[7]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[8]  Florian Schmidt-Weigand,et al.  A closer look at split visual attention in system- and self-paced instruction in multimedia learning , 2010 .

[9]  David R. Olson,et al.  Memory and inference in the comprehension of oral and written discourse , 1978 .

[10]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[11]  Paul Ginns.  Meta-Analysis of the Modality Effect. , 2005 .

[12]  J. Sweller,et al.  Cognitive load theory, modality of presentation and the transient information effect. , 2011 .

[13]  M. Mayrath,et al.  Varying Tutorial Modality and Interface Restriction to Maximize Transfer in a Complex Simulation Environment. , 2011 .

[14]  Richard E. Mayer,et al.  Principles for managing essential processing in multimedia learning: Segmenting, pre-training, and modality principles. , 2005 .

[15]  J. Sweller,et al.  Reducing cognitive load by mixing auditory and visual presentation modes , 1995 .

[16]  R. Green.  REMEMBERING IDEAS FROM TEXT: THE EFFECT OF MODALITY OF PRESENTATION , 1981 .

[17]  Slava Kalyuga,et al.  Incorporating Learner Experience into the Design of Multimedia Instruction. , 2000 .

[18]  J. Sweller.  Implications of Cognitive Load Theory for Multimedia Learning , 2005, The Cambridge Handbook of Multimedia Learning.

[19]  Huib K. Tabbers,et al.  Where did the Modality Principle in Multimedia Learning Go? A ­Double Replication Failure that Questions Both Theory and Practical Use , 2011 .

[20]  L. R. Peterson,et al.  Short-term retention of individual verbal items. , 1959, Journal of experimental psychology.

[21]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[22]  G. Rickheit,et al.  Modalitätsspezifische Textverarbeitung bei Personen mit unterschiedlichen Kommunikationsschwerpunkten , 1987 .

[23]  C. Perfetti,et al.  Learning to read : basic research and its implications , 1993 .

[24]  Roland Brünken,et al.  Working Memory Interference During Processing Texts and Pictures: Implications for the Explanation of the Modality Effect , 2010 .

[25]  R. Folger,et al.  When is it Acceptable to Accept a Null Hypothesis: No Way, Jose? , 1998 .

[26]  Janet Kleber,et al.  On the Robustness of the Modality ­Effect: Attempting to Replicate a Basic Finding , 2011 .

[27]  Klaus D. Stiller,et al.  How pacing of multimedia instructions can influence modality effects: A case of superiority of visual texts , 2009 .

[28]  Katharina Scheiter,et al.  Does the Modality Effect in Multimedia Learning Appear only with Text Containing Spatial Information , 2011 .

[29]  Sharon K Tindall-Ford,et al.  When two sensory modes are better than one , 1997 .

[30]  D. Rubin,et al.  Reading and listening to oral‐based versus literate‐based discourse , 2000 .

[31]  V. Gyselinck,et al.  The role of working memory components in multimedia comprehension , 2008 .

[32]  Florian Schmidt-Weigand,et al.  Does Animation Amplify the Modality Effect – or is there any Modality Effect at All? , 2011 .

[33]  Florian Schmidt-Weigand,et al.  Explaining the modality and contiguity effects: New insights from investigating students' viewing behaviour , 2010 .

[34]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[35]  R. Mayer,et al.  Multimedia Learning: Frontmatter , 2001 .

[36]  W. Schnotz,et al.  Individuelle Modalitätspräferenzen beim Textverstehen , 2005 .

[37]  Richard K. Lowe,et al.  A unified view of learning from animated and static graphics , 2008 .

[38]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[39]  A. Baddeley.  Essentials of Human Memory , 1999 .

[40]  M. Just,et al.  Constructing mental models of machines from text and diagrams. , 1993 .

[41]  Christian Kürschner,et al.  Das Verhältnis gesprochener und geschriebener Sprache bei der Konstruktion mentaler Repräsentationen , 2008 .

[42]  R. Mayer,et al.  Cognitive Principles of Multimedia Learning: The Role of Modality and Contiguity , 1999 .

[43]  W. Howard Levie,et al.  Effects of text illustrations: A review of research , 1982 .