We present here some transmedia similarity measures that we recently designed by adopting some "intermediate level" fusion approaches. The main idea is to use some principles coming from pseudo-relevance feedback and, more specifically, transmedia pseudo-relevance feedback for enriching the mono-media representation of an object with features coming from the other media. One issue that arises when adopting such a strategy is to determine how to compute the mono-media similarity between an aggregate of objects coming from a first (pseudo-)feedback step and one single multimodal object. We propose two alternative ways of adressing this issue, that result in what we called the "transmedia document reranking" and "complementary feedback" methods respectively.
For the ImageCLEF - Photo Retrieval Task, it appears that mono-media retrieval performance is more or less equivalent for pure image and pure text content (around 20% MAP). Using our transmedia pseudofeedback-based similarity measures allowed us to dramatically increase the performance by ~50% (relative). From a cross-lingual perspective, the use of domain-specific, corpus-adapted probabilistic dictionaries seems to offer better results than the use of a broader, more general standard dictionary. With respect to the monolingual baselines, multilingual runs show a slight degradation of retrieval performance ( ~6 to 10% relative).
[1]
Joo-Hwee Lim,et al.
IPAL Inter-Media Pseudo-Relevance Feedback Approach to ImageCLEF 2006 Photo Retrieval
,
2006,
CLEF.
[2]
Joo-Hwee Lim,et al.
Inter-media Pseudo-relevance Feedback Application to ImageCLEF 2006 Photo Retrieval
,
2006,
CLEF.
[3]
Hsin-Hsi Chen,et al.
Approaches Using a Word-Image Ontology and an Annotated Image Corpus as Intermedia for Cross-Language Image Retrieval
,
2006,
CLEF.
[4]
Florent Perronnin,et al.
Fisher Kernels on Visual Vocabularies for Image Categorization
,
2007,
2007 IEEE Conference on Computer Vision and Pattern Recognition.
[5]
Gabriela Csurka,et al.
XRCE's Participation in ImageCLEF 2009
,
2009,
CLEF.
[6]
Gabriela Csurka,et al.
XRCE's Participation to ImageCLEF 2008
,
2008,
CLEF.
[7]
R. Manmatha,et al.
A Model for Learning the Semantics of Pictures
,
2003,
NIPS.
[8]
John D. Lafferty,et al.
Model-based feedback in the language modeling approach to information retrieval
,
2001,
CIKM '01.
[9]
Allan Hanbury,et al.
Overview of the ImageCLEFphoto 2007 Photographic Retrieval Task
,
2008,
CLEF.