Questions, pictures, answers : Introducing pictures in question-answering systems

We present the Dutch IMIX research programme on multimodal interaction, speech and language technology. We discuss our contributions to this programme in the form of two research projects, IMOGEN and VIDIAM, and the technical integration of the various modules developed by IMIX subprojects to build a demonstrator that shows the project’s aims and achievements. In this paper we pay special attention to the role that images could play in iterative question answering.

[1]  Emiel Krahmer,et al.  Normalized alignment of dependency trees for detecting textual entailment , 2006 .

[2]  Russell N. Carney,et al.  Pictorial Illustrations Still Improve Students' Learning from Text , 2002 .

[3]  Masashi Inoue.  On the need for annotation-based image retrieval , 2004 .

[4]  Michelle X. Zhou,et al.  A graph-matching approach to dynamic media allocation in intelligent multimedia interfaces , 2005, IUI '05.

[5]  Chris Callison-Burch,et al.  Paraphrase Substitution for Recognizing Textual Entailment , 2006, CLEF.

[6]  E. Krahmer,et al.  Information Modalities for Procedural Instructions: The Influence of Text, Pictures, and Film Clips on Learning and Executing RSI Exercises , 2008, IEEE Transactions on Professional Communication.

[7]  Emiel Krahmer,et al.  Explorations in Sentence Fusion , 2005, ENLG.

[8]  Harry Bunt,et al.  From question answering to spoken dialogue: towards an information search assistant for interactive multimodal information extraction , 2005, INTERSPEECH.

[9]  Pat Hanrahan,et al.  Identification and validation of cognitive design principles for automated generation of assembly instructions , 2004, AVI.

[10]  W. E. Bosma.  Image Retrieval Supports Multimedia Authoring , 2005 .

[11]  Sanda M. Harabagiu,et al.  Experiments with Interactive Question Answering in Complex Scenarios , 2004, Workshop On Pragmatics Of Question Answering.

[12]  Jörg Tiedemann,et al.  Question Answering for Dutch using Dependency Relations , 2005, CLEF.

[13]  李幼升,et al.  Ph , 1989 .

[14]  Anna Rumshisky,et al.  Discourse Structure of Text-Graphics Documents , 2001 .

[15]  W. E. Bosma.  Query-based extracting: how to support the answer? , 2006 .

[16]  Niels Ole Bernsen,et al.  Foundations of Multimodal Representations: A Taxonomy of Representational Modalities , 1994, Interact. Comput..

[17]  Gerd Herzog,et al.  MULTIPLATFORM Testbed: An Integration Platform for Multimodal Dialog Systems , 2003, HLT-NAACL 2003.

[18]  Lou Boves,et al.  Longer-length acoustic units for continuous speech recognition , 2005, 2005 13th European Signal Processing Conference.

[19]  Bettina Berendt,et al.  Conceptual Taxonomy Identification in Medical Documents , 2005 .

[20]  Rieks op den Akker,et al.  Follow-up utterances in QA dialogue , 2006 .

[21]  William C. Mann,et al.  Rhetorical Structure Theory: Toward a functional theory of text organization , 1988 .

[22]  Marilyn Domas White,et al.  A taxonomy of relationships between images and text , 2003, J. Documentation.

[23]  Suresh Manandhar,et al.  Implementing clarification dialogues in open domain question answering , 2005, Natural Language Engineering.

[24]  R. Mayer,et al.  Animation as an Aid to Multimedia Learning , 2002 .

[25]  Mark Andrew Greenwood,et al.  Open-domain question answering , 2005 .

[26]  Eduard H. Hovy,et al.  On the Knowledge Underlying Multimedia Presentations , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[27]  Tsuneaki Kato,et al.  Working notes of the fourth NTCIR workshop meeting , 2004 .

[28]  Gosse Bouma,et al.  Developing Offline Strategies for Answering Medical Questions , 2005 .

[29]  W. E. Bosma.  Extending Answers using Discourse Structure , 2005 .

[30]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[31]  John M. Prager,et al.  Open-Domain Question-Answering , 2007, Found. Trends Inf. Retr..

[32]  Elisabeth André,et al.  Ein planbasierter Ansatz zur Generierung multimedialer Präsentationen , 1995, DISKI.

[33]  Noriko Kando,et al.  WoZ Simulation of Interactive Question Answering , 2006, HLT-NAACL 2006.

[34]  Harry Bunt,et al.  Multidimensional Dialogue Management , 2006, SIGDIAL Workshop.

[35]  Yan Han,et al.  Trajectory clustering for automatic speech recognition , 2005, 2005 13th European Signal Processing Conference.

[36]  Olivier Galibert,et al.  Ritel: an open-domain, human-computer dialog system , 2005, INTERSPEECH.

[37]  Junichi Fukumoto,et al.  Rits-QA: List answer detection and Context task with ellipses handling , 2004 .