On the Role of Visuals in Multimodal Answers to Medical Questions

This paper describes two experiments carried out in order to investigate the role of visuals in multimodal answer presentations for a medical question answering system. First, a production experiment was carried out to determine which modalities people choose to answer different types of questions. In this experiment, participants had to create (multimodal) presentations of answers to general medical questions. The collected answer presentations were coded on the presence of visual media (i.e., photos, graphics, and animations) and their function. The results indicated that participants presented the information in a multimodal way. Moreover, significant differences were found in the presentation of different answer and question types. Next, an evaluation experiment was conducted to investigate how users evaluate different types of multimodal answer presentations. In this second experiment, participants had to assess the informativity and attractiveness of answer presentations for different types of medical questions. These answer presentations, originating from the production experiment, were manipulated in their answer length (brief vs. extended) and their type of picture (illustrative vs. informative). After the participants had assessed the answer presentations, they received a post- test in which they had to indicate how much they had recalled from the presented answer presentations. The results showed that answer presentations with an informative picture were evaluated as more informative and more attractive than answer presentations with an illustrative picture. The results for the post-test tentatively indicated that learning from answer presentations with an informative picture leads to a better learning performance than learning from purely textual answer presentations.

[1]  J. Hedberg.  The Cambridge handbook of multimedia learning , 2007 .

[2]  Rachelle S. Heller,et al.  Using a theoretical multimedia taxonomy framework , 2001, JERC.

[3]  Alistair G. Sutcliffe,et al.  Task-related information analysis , 1997, Int. J. Hum. Comput. Stud..

[4]  Eduard H. Hovy,et al.  On the Knowledge Underlying Multimedia Presentations , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[5]  Dianne C. Berry,et al.  Learning a procedural task: effectiveness of multimedia presentations , 2000 .

[6]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[7]  J. Sweller,et al.  Understanding Instructions , 2004 .

[8]  Mark T. Maybury,et al.  15. Multimedia & Multimodal Interaction Structure , 2000 .

[9]  W. E. Bosma.  Extending Answers using Discourse Structure , 2005 .

[10]  Anton Nijholt,et al.  Questions, pictures, answers : Introducing pictures in question-answering systems , 2007 .

[11]  Sharon L. Oviatt,et al.  Toward a theory of organized multimodal integration patterns during human-computer interaction , 2003, ICMI '03.

[12]  Russell N. Carney,et al.  Pictorial Illustrations Still Improve Students' Learning from Text , 2002 .

[13]  E. André.  The Generation of Multimedia Presentations , 2000 .

[14]  E. Krahmer,et al.  Information Modalities for Procedural Instructions: The Influence of Text, Pictures, and Film Clips on Learning and Executing RSI Exercises , 2008, IEEE Transactions on Professional Communication.

[15]  Pat Hanrahan,et al.  Identification and validation of cognitive design principles for automated generation of assembly instructions , 2004, AVI.

[16]  K. Krippendorff.  Krippendorff, Klaus, Content Analysis: An Introduction to its Methodology . Beverly Hills, CA: Sage, 1980. , 1980 .

[17]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[18]  Toni Rietveld,et al.  Statistical Techniques for the Study of Language and Language Behaviour , 1993 .

[19]  Joan K. Gallini,et al.  When Is an Illustration Worth Ten Thousand Words , 1990 .

[20]  Emiel Krahmer,et al.  Towards automatic generation of multimodal answers to medical questions: a cognitive engineering approach , 2007 .

[21]  Barbara Tversky,et al.  Animation: can it facilitate? , 2002, Int. J. Hum. Comput. Stud..

[22]  R. Mayer.  Systematic Thinking Fostered by Illustrations in Scientific Text , 1989 .

[23]  Antonio Moreno-Sandoval,et al.  CROSSING BARRIERS IN TEXT SUMMARIZATION RESEARCH , 2005 .

[24]  R. Mayer,et al.  Aids to computer-based multimedia learning , 2002 .

[25]  S. Schwan,et al.  The cognitive benefits of interactive videos: learning to tie nautical knots , 2004 .