Semantic-Friendly Indexing and Quering of Images Based on the Extraction of the Objective Semantic Cues

Abstract image semantics resists all forms of modeling, very much like any kind of intelligence does. However, in order to develop more satisfying image navigation systems, we need tools to construct a semantic bridge between the user and the database. In this paper we present an image indexing scheme and a query language, which allow the user to introduce cognitive dimension to the search. At an abstract level, this approach consists of: (1) learning the “natural language” that humans speak to communicate their semantic experience of images, (2) understanding the relationships between this language and objective measurable image attributes, and then (3) developing corresponding feature extraction schemes.More precisely, we have conducted a number of subjective experiments in which we asked human subjects to group images, and then explain verbally why they did so. The results of this study indicated that a part of the abstraction involved in image interpretation is often driven by semantic categories, which can be broken into more tangible semantic entities, i.e. objective semantic indicators. By analyzing our experimental data, we have identified some candidate semantic categories (i.e. portraits, people, crowds, cityscapes, landscapes, etc.) and their underlying semantic indicators (i.e. skin, sky, water, object, etc.). These experiments also helped us derive important low-level image descriptors, accounting for our perception of these indicators.We have then used these findings to develop an image feature extraction and indexing scheme. In particular, our feature set has been carefully designed to match the way humans communicate image meaning. This led us to the development of a “semantic-friendly” query language for browsing and searching diverse collections of images.We have implemented our approach into an Internet search engine, and tested it on a large number of images. The results we obtained are very promising.

[1]  H. Barlow.  Vision Science: Photons to Phenomenology by Stephen E. Palmer , 2000, Trends in Cognitive Sciences.

[2]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Aleksandra Mojsilovic,et al.  Capturing image semantics with low-level descriptors , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[4]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[5]  M. Wertheimer.  Untersuchungen zur Lehre von der Gestalt. II , 1923 .

[6]  Aleksandra Mojsilovic,et al.  The vocabulary and grammar of color patterns , 2000, IEEE Trans. Image Process..

[7]  Jianying Hu,et al.  Matching and retrieval based on the vocabulary and grammar of color patterns , 2000, IEEE Trans. Image Process..

[8]  Alberto Del Bimbo,et al.  Image retrieval by color semantics , 1999, Multimedia Systems.

[9]  G. J. Hahn,et al.  Statistical models in engineering , 1967 .

[10]  Gunther Wyszecki,et al.  Color Science: Concepts and Methods, Quantitative Data and Formulae, 2nd Edition , 2000 .

[11]  B. S. Manjunath,et al.  NeTra: A toolbox for navigating large image databases , 1997, Multimedia Systems.

[12]  K. L. Kelly.  The ISCC-NBS method of designating colors and a dictionary of color names , 1955 .

[13]  James Ze Wang,et al.  System for screening objectionable images , 1998, Comput. Commun..

[14]  M. Paul.  A dictionary of color , 1930 .

[15]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[16]  N. Cox.  Statistical Models in Engineering , 1970 .

[17]  Tony Belpaeme,et al.  Factors influencing the origins of colour categories , 2002 .

[18]  Simone Santini.  MIXED MEDIA SEARCH IN IMAGE DATABASES USING TEXT AND VISUAL SIMILARITY , 2001 .

[19]  Anil K. Jain,et al.  On image classification: city vs. landscape , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[20]  J. Lammens.  A computational model of color perception and color naming , 1995 .

[21]  Thomas S. Huang,et al.  Content-based image retrieval with relevance feedback in MARS , 1997, Proceedings of International Conference on Image Processing.

[22]  Anil K. Jain,et al.  Shape-Based Retrieval: A Case Study With Trademark Image Databases , 1998, Pattern Recognit..

[23]  A. Witkin,et al.  On the Role of Structure in Vision , 1983 .

[24]  Aleksandra Mojsilovic.  A method for color naming and description of color composition in images , 2002, Proceedings. International Conference on Image Processing.

[25]  Dorin Comaniciu,et al.  Mean shift analysis and applications , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[26]  Martin Szummer,et al.  Indoor-outdoor image classification , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[27]  Charles A. Bouman,et al.  Perceptual image similarity experiments , 1998, Electronic Imaging.

[28]  Simone Santini.  Query paradigm to discover the relation between text and images , 2001, IS&T/SPIE Electronic Imaging.

[29]  Aleksandra Mojsilovic,et al.  A Variational Approach to Recovering a Manifold from Sample Points , 2002, ECCV.

[30]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[31]  Ramesh C. Jain,et al.  A Visual Information Management System for the Interactive Retrieval of Faces , 1993, IEEE Trans. Knowl. Data Eng..

[32]  S. Osher,et al.  Level set methods: an overview and some recent results , 2001 .

[33]  Wei Zhu,et al.  Image organization and retrieval using a flexible shape model , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[34]  J. Koenderink.  The structure of images , 2004, Biological Cybernetics.

[35]  Olivier D. Faugeras,et al.  Co-dimension 2 Geodesic Active Contours for MRA Segmentation , 1999, IPMI.

[36]  David A. Forsyth,et al.  Body plans , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  H. Soner,et al.  Level set approach to mean curvature flow in arbitrary codimension , 1996 .

[38]  P. Kay,et al.  Basic Color Terms: Their Universality and Evolution , 1973 .

[39]  William Schroeder,et al.  The Visualization Toolkit: An Object-Oriented Approach to 3-D Graphics , 1997 .

[40]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[41]  Georgios Tziritas,et al.  Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis , 1999, IEEE Trans. Multim..

[42]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[44]  S. Zucker.  Computational and Psychophysical Experiments in Grouping: Early Orientation Selection , 1983 .

[45]  B. S. Manjunath,et al.  NeTra: A toolbox for navigating large image databases , 1997, Proceedings of International Conference on Image Processing.

[46]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[47]  Anna O. Shepard,et al.  The ISCC-NBS Method of Designating Colors and a Dictionary of Color Names. Kenneth L. Kelley and Deane B. Judd National Bureau of Standards, Department of Commerce, Washington, 1955. v + 158 pp., 12 figs. $2.00. , 1957, American Antiquity.

[48]  David A. Forsyth,et al.  Finding Naked People , 1996, ECCV.

[49]  M. Grayson.  The heat equation shrinks embedded plane curves to round points , 1987 .

[50]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Wayne H. Wolf,et al.  Semantic image retrieval through human subject segmentation and characterization , 1997, Electronic Imaging.

[52]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[53]  Ioannis Pitas,et al.  Extraction of facial regions and features using color and shape information , 1996, Proceedings of 13th International Conference on Pattern Recognition.