Modality effects on cognitive load and performance in high-load information presentation

In this study, we argue that modality planning in multimodal presentation systems needs to consider the modality characteristics at not only the presentational level but also the cognitive level, especially in a situation where the information load is high and the user task is time-critical. As a first step towards automatic cognitive-aware modality planning, we integrated the effect of different modalities on cognitive load and performance, using a high-load information presentation scenario. Mainly based on modality-related psychology theories, we selected five modality conditions (text, image, text+image, text+speech, and text+sound) and made hypotheses about their effects on cognitive load. Modality effects were evaluated by two cognitive load measurements and two performance measurements. Results confirmed most of the predicted modality effects, and showed that these effects become significant when the information load and the task demand are high. The findings of this study suggest that it is highly necessary to encode modality-related principles of human cognition into the modality planning procedure for systems that support high-load human-computer interaction.

[1]  Xiang Yao,et al.  The Design of a Dynamic Emergency Response Management Information System (DERMIS) , 2004 .

[2]  Anton Nijholt,et al.  Towards a Unified Knowledge-Based Approach to Modality Choice , 2007 .

[3]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[4]  Graham Wilson,et al.  Psychophysiological assessment of workload in multi-task environments , 2020 .

[5]  Noureddine Elouazizi,et al.  On cognitive relevance in automatic multimodal systems , 2004, IEEE Sixth International Symposium on Multimedia Software Engineering.

[6]  Steven K. Feiner,et al.  Automating the generation of coordinated multimedia explanations , 1991, Computer.

[7]  R. Mayer,et al.  Nine Ways to Reduce Cognitive Load in Multimedia Learning , 2003 .

[8]  H. Kirchmann,et al.  SmartKom : Symmetric Multimodality in an Adaptive and Reusable Dialogue Shell , 2003 .

[9]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[10]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[11]  D. Leutner,et al.  Direct Measurement of Cognitive Load in Multimedia Learning , 2003 .

[12]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[13]  Jonathan Grudin,et al.  Speech, language, and audition , 1995 .

[14]  Mohammed Yeasin,et al.  Speech-gesture driven multimodal interfaces for crisis management , 2003, Proc. IEEE.

[15]  E. André.  The Generation of Multimedia Presentations , 2000 .

[16]  Susanne van Mulken,et al.  Structure and Rules in Automated Multimedia Presentation Planning , 1993, IJCAI.

[17]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[18]  J G Hollands,et al.  ENGINEERING PSYCHOLOGY AND HUMAN PERFORMANCE - THIRD EDITION , 2000 .

[19]  Niels Ole Bernsen,et al.  Multimodality in Language and Speech Systems — From Theory to Design Support Tool , 2002 .

[20]  E. André,et al.  WIP: The Coordinated Generation of Multimodal Presentations from a Common Representation , 1992 .

[21]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[22]  A. Paivio,et al.  Dual coding theory and education , 1991 .

[23]  J. M. Peter,et al.  The Efficacy of Psychophysiological Measures for Implementing Adaptive Technology , 2001 .

[24]  P. Chandler,et al.  Cognitive Load Theory and the Format of Instruction , 1991 .