Conceptual framework for indexing visual information at multiple levels

In this paper, we present a conceptual framework for indexing different aspects of visual information. Our framework unifies concepts from this literature in diverse fields such as cognitive psychology, library sciences, art, and the more recent content-based retrieval. We present multiple level structures for visual and non-visual and non- visual information. The ten-level visual structure presented provides a systematic way of indexing images based on syntax and semantics, and includes distinctions between general concept and visual concept. We define different types of relations at different levels of the visual structure, and also use a semantic information table to summarize important aspects related to an image. While the focus is on the development of a conceptual indexing structure, our aim is also to bring together the knowledge from various fields, unifying the issues that should be considered when building a digital image library. Our analysis stresses the limitations of state of the art content-based retrieval systems and suggests areas in which improvements are necessary.

[1]  Anil K. Jain,et al.  On image classification: city vs. landscape , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[2]  Barbara Orbach.  So That Others May See: Tools for Cataloging Still Images. , 1990 .

[3]  James M. Turner,et al.  Determining the subject content of still and moving image documents for storage and retrieval : an experimental investigation , 1994 .

[4]  Corinne Jörgensen,et al.  Indexing Images: Testing an Image Description Template. , 1996 .

[5]  Corinne Jörgensen,et al.  Multiple Level Classification of Visual Descriptors in the Generic AV DS , 1999 .

[6]  Sara Shatford,et al.  Analyzing the Subject of a Picture: A Theoretical Approach , 1986 .

[7]  Martin Szummer,et al.  Indoor-outdoor image classification , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[8]  Shih-Fu Chang,et al.  Visual information retrieval from large distributed online repositories , 1997, CACM.

[9]  Shih-Fu Chang,et al.  Model-based classification of visual information for content-based retrieval , 1998, Electronic Imaging.

[10]  R. Arnheim.  Art and visual perception: A psychology of the creative eye, New version , 1955 .

[11]  Bing Liu,et al.  Content-based Retrieval , 2009, Encyclopedia of Database Systems.

[12]  K. Markey.  Computer-Assisted Construction of a Thematic Catalog of Primary and Secondary Subject Matter , 1983 .

[13]  B. Burns.  Percepts, concepts, and categories : the representation and processing of information , 1992 .

[14]  Erwin Panofsky,et al.  Studies in Iconology , 1962 .

[15]  Shih-Fu Chang,et al.  Image and video search engine for the World Wide Web , 1997, Electronic Imaging.

[16]  Sara Shatford Layne,et al.  Some Issues in the Indexing of Images , 1994, J. Am. Soc. Inf. Sci..

[17]  Anne Treisman,et al.  Preattentive processing in vision , 1985, Computer Vision Graphics and Image Processing.

[18]  A. Tversky.  Features of Similarity , 1977 .

[19]  Shih-Fu Chang,et al.  Automatic selection of visual features and classifiers , 1999, Electronic Imaging.

[20]  Shih-Fu Chang,et al.  Integrating Multiple Classifiers In Visual Object Detectors Learned From User Input , 2000 .

[21]  Daniel Hernández,et al.  Qualitative Representation of Spatial Knowledge , 1994, Lecture Notes in Computer Science.

[22]  D. Ellis.  Visual explanations: Images and quantities , 1997 .

[23]  J. P. Eakins.  Design criteria for a shape retrieval system , 1993 .

[24]  Ahmed Karmouch,et al.  A Temporal Model for Interactive Multimedia Scenarios , 1995, IEEE Multim..

[25]  Sylvan Barnet,et al.  A Short Guide to Writing about Art , 1985 .

[26]  Nina Wacholder,et al.  Disambiguation of Proper Names in Text , 1997, ANLP.

[27]  E. Rosch,et al.  Family resemblances: Studies in the internal structure of categories , 1975, Cognitive Psychology.

[28]  S. Harnad.  Categorical Perception: The Groundwork of Cognition , 1990 .

[29]  B. Burns.  6 Perceived Similarity in Perceptual and Conceptual Development: The Influence of Category Information on Perceptual Organization , 1992 .

[30]  James M. Turner.  Cross-Language Transfer of Indexing Concepts for Storage and Retrieval of Moving Images: Preliminary Results. , 1996 .

[31]  A. M. Triesman,et al.  Preattentive processing in vision , 1985 .

[32]  Helene Roberts,et al.  "DO YOU HAVE ANY PICTURES OF.....?": SUBJECT ACCESS TO WORKS OF ART IN VISUAL COLLECTIONS AND BOOK REPRODUCTIONS , 1988, Art Documentation: Journal of the Art Libraries Society of North America.

[33]  Shih-Fu Chang,et al.  Integration of Visual and Text-Based Approaches for the Content Labeling and Classification of Photographs , 1999, SIGIR 1999.

[34]  Eric T. Davis,et al.  A Prototype Item-Level Index to the Civil War Photograph Collection of the Ohio Historical Society. , 1997 .

[35]  Kai-Uwe Carstensen,et al.  Modelling Spatial Knowledge on a Linguistic Basis , 1990, Lecture Notes in Computer Science.

[36]  Neff Walker,et al.  A classification of visual representations , 1994, CACM.

[37]  Corinne Jörgensen.  Classifying Images: Criteria for Grouping as Revealed in a Sorting Task , 1995 .

[38]  H. Purchase,et al.  Defining multimedia , 1998 .

[39]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[40]  G. Murphy,et al.  Converging operations on a basic level in event taxonomies , 1990, Memory & cognition.

[41]  Simone Santini,et al.  Beyond query by example , 1998, 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No.98EX175).

[42]  J. Davenport.  Editor , 1960 .

[43]  Linda B. Smith,et al.  Perceptual Similarity and Conceptual Structure , 1992 .

[44]  Donis A. Dondis.  A primer of visual literacy , 1973 .

[45]  A. Rifkin,et al.  Evidence for a basic level in event taxonomies , 1985, Memory & cognition.

[46]  Raya Fidel,et al.  Challenges in Indexing Electronic Text and Images , 1994 .

[47]  C. Martin,et al.  A Media Taxonomy , 1995, IEEE Multim..

[48]  Sharon Lee Armstrong,et al.  What some concepts might not be , 1983, Cognition.

[49]  Beverly J. Jones,et al.  Variability and Universality in Human Image Processing , 1995 .

[50]  Simone Santini,et al.  Gabor space and the development of preattentive similarity , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[51]  Shih-Fu Chang,et al.  AMOS: an active system for MPEG-4 video object segmentation , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[52]  Craig A. Lindley,et al.  A Multi-Model Framework for Video Information Systems , 1999, DS-8.

[53]  Wayne D. Gray,et al.  Basic objects in natural categories , 1976, Cognitive Psychology.

[54]  R. C. Langford.  How People Look at Pictures, A Study of the Psychology of Perception in Art. , 1936 .

[55]  Thierry Pun,et al.  A Comparison of Human and Machine Assessments of Image Similarity for the Organization of Image Databases , 1997 .

[56]  B. Tversky,et al.  Categories of environmental scenes , 1983, Cognitive Psychology.

[57]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[58]  Bella Hass Weinberg,et al.  Challenges in indexing electronic text and images , 1994 .

[59]  Elisabeth Betz Parker,et al.  LC thesaurus for graphic materials : topical terms for subject access , 1987 .