Mutual text-image queries

This paper presents a novel approach to support students to learn a comprehensive domain-specific terminology and to understand textual descriptions of complex-shaped objects. We implemented an experimental system where learners can interactively explore textual descriptions and 3D visualizations. We propose a method for hierarchical content representations of text documents and views on 3D models. Based on these data structures, user interactions on texts and interactive 3D visualizations are transformed into queries to an information retrieval system. This enables us to coordinate the content of both media, to focus the attention of the user on the most salient graphical objects, and to suggest potential relevant text segments in large text documents and appropriate views on 3D models to illustrate the spatial relations between the relevant domain objects of the query. Finally, we demonstrated this concept in an interactive tutoring environment based on standard textbooks on human anatomy.

[1]  Edward L. Robertson,et al.  Nonlinear magnification fields , 1997, Proceedings of VIZ '97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium.

[2]  L. Brooke.  The National Library of Medicine. , 1980, Hospital libraries.

[3]  Pere-Pau Vázquez,et al.  Correlating Text and Images: Concept and Evaluation , 2007, Smart Graphics.

[4]  Wolfgang Wahlster,et al.  Plan-Based Integration of Natural Language and Graphics Generation , 1993, Artif. Intell..

[5]  Mateu Sbert,et al.  Viewpoint Selection using Viewpoint Entropy , 2001, VMV.

[6]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[7]  G. Tortora.  Introduction to the Human Body: The Essentials of Anatomy and Physiology , 1997 .

[8]  Tobias Isenberg,et al.  User-Supported Interactive Illustration of Text , 2005, SimVis.

[9]  G. W. Furnas,et al.  Generalized fisheye views , 1986, CHI '86.

[10]  Thomas Ertl,et al.  Transparency in Interactive Technical Illustrations , 2002, Comput. Graph. Forum.

[11]  M. Sheelagh T. Carpendale,et al.  A framework for unifying presentation space , 2001, UIST '01.

[12]  G Salton,et al.  Automatic Analysis, Theme Generation, and Summarization of Machine-Readable Texts , 1994, Science.

[13]  Timo Götzelmann,et al.  SpringLens Distributed Nonlinear Magnifications , 2006, Eurographics.

[14]  Jim X. Chen,et al.  Nonlinear perspective projections and magic lenses: 3D view deformation , 2005, IEEE Computer Graphics and Applications.

[15]  Henry Lieberman,et al.  Beating Common Sense into Interactive Applications , 2004, AI Mag..

[16]  Satoru Kawai,et al.  A simple method for computing general position in displaying three-dimensional objects , 1988, Comput. Vis. Graph. Image Process..

[17]  Timo Götzelmann,et al.  Annotating Images through Adaptation: An Integrated Text Authoring and Illustration Framework , 2007, J. WSCG.

[18]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[19]  K. Hartmann,et al.  A spreading activation approach to text illustration , 2002, SMARTGRAPH '02.

[20]  Stefan Schlechtweg,et al.  Generating Scientific Illustrations in Digital Books , 2000 .

[21]  J. Laidlaw,et al.  ANATOMY OF THE HUMAN BODY , 1967, The Ulster Medical Journal.

[22]  Timo Götzelmann,et al.  Agent-Based Annotation of Interactive 3D Visualizations , 2006, Smart Graphics.

[23]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[24]  Steven K. Feiner,et al.  Automating the generation of coordinated multimedia explanations , 1991, Computer.

[25]  Stefan Bruckner,et al.  TECHNICAL REPORT VolumeShop: An Interactive System for Direct Volume , 2022 .

[26]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[27]  Michael John Ackerman,et al.  The Visible Human Project. , 1991 .

[28]  Mateu Sbert,et al.  Importance-Driven Focus of Attention , 2006, IEEE Transactions on Visualization and Computer Graphics.

[29]  Thomas Strothotte,et al.  Illustrative Browsing: A New Method of Browsing in Long On-line Texts , 1999, INTERACT.

[30]  Mateu Sbert,et al.  Realtime automatic selection of good molecular views , 2006, Comput. Graph..

[31]  Ivan Viola,et al.  Importance-driven volume rendering , 2004, IEEE Visualization 2004.