An operational model based on knowledge representation for querying the image content with concepts and relations

In order to overcome the semantic gap (i.e. the gap between low-level extracted features and semantic description in state-of-the-art content-based image retrieval systems, a class of frameworks proposed within the framework of the European Fermi project, consisted of modeling the semantic content of images following a sharp process of human-assisted indexing. These approaches, based on expressive knowledge-based representation models provide satisfactory results in terms of retrieval quality but are not easily usable on large collections of images because of the necessary human intervention required for indexing. We propose in this paper to integrate the content-based and semantic-based solutions through a model featuring semantic and relational characterizations of the multimedia (image) content for automatic symbolic indexing and retrieval. Its instantiation as an image retrieval framework relies on a representation formalism handling high-level image descriptions and allowing to query with conceptual descriptors. Our approach complements content-based solutions through the mapping of low-level extracted features to semantic concepts and the manipulation of graph-based symbolic index and query structures; and extends the semantic-based solutions by considering automatically-extracted semantic and relational information. At the experimental level, we evaluate the retrieval performance of our system on queries coupling both semantic and relational characterizations through recall and precision indicators on a test collection of 2,500 color photographs and the TRECVID keyframe corpus.