Query paradigm to discover the relation between text and images

This paper studies the relation between images and text in image databases. An analysis of this relation results in the definition of three distinct query modalities: (1) linguistic scenario: images are part of a whole including a self-contained linguistic discourse, and their meaning derives form their interaction with the linguistic discourse. A typical case of this scenario is constituted by images on the World Wide Web; (2) closed world scenario: images are defined in a limited domain, and their meaning is anchored by conventions and norms in that domain; (3) user scenario: the linguistic discourse is provided by the user. This is the case of highly interactive systems with relevance feedback. This paper deals with image databases of the first type. It shows how the relation between images and text can be inferred, and exploited for search. The paper develops a similarity model in which the similarity between two images is given by both their visual similarity and the similarity of the attached words. Both the visual and textural similarity can be manipulated by the user through the two windows of the interface.

[1]  Umberto Eco,et al.  A theory of semiotics , 1976, Advances in semiotics.

[2]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[3]  Prabhakar Raghavan,et al.  Mining the Link Structure of the World Wide Web , 1998 .

[4]  Paul H. Lewis,et al.  Semiotics and agents for integrating and navigating through multimedia representations of concepts , 1999, Electronic Imaging.

[5]  L. R. Rasmussen,et al.  In information retrieval: data structures and algorithms , 1992 .

[6]  Simone Santini.  Semantic modalities in content-based retrieval , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[7]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Einat Amitay,et al.  Using common hypertext links to identify the best phrasal description of target web documents , 1998 .

[9]  Ronald Fagin,et al.  Combining Fuzzy Information from Multiple Systems , 1999, J. Comput. Syst. Sci..

[10]  Shih-Fu Chang,et al.  Visually Searching the Web for Content , 1997, IEEE Multim..

[11]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[12]  Jon M. Kleinberg,et al.  Automatic Resource Compilation by Analyzing Hyperlink Structure and Associated Text , 1998, Comput. Networks.

[13]  Markus A. Stricker,et al.  Similarity of color images , 1995, Electronic Imaging.