Semantic keyword-based retrieval of photos taken with mobile devices

This paper presents an approach for incorporating contextual metadata in a keyword-based photo retrieval process. We use our mobile annotation system PhotoMap in order to create metadata describing the photo shoot context (e.g., street address, nearby objects, season, lighting, nearby people...). These metadata are then used to generate a set of stamped words for indexing each photo. We adapt the Vector Space Model (VSM) in order to transform these shoot context words into document-vector terms. Furthermore, spatial reasoning is used for inferring new potential indexing terms. We define methods for weighting those terms and for handling a query matching. We also detail retrieval experiments carried out by using PhotoMap and Flickr geotagged photos. We illustrate the advantages of using Wikipedia georeferenced objects for indexing photos.

[1]  James A. Hendler.  Web 3.0: Chicken Farms on the Semantic Web , 2008, Computer.

[2]  Lei Zhang,et al.  IGroup: presenting web image search results in semantic clusters , 2007, CHI.

[3]  Fergal Monaghan,et al.  Automating Photo Annotation using Services and Ontologies , 2006, 7th International Conference on Mobile Data Management (MDM'06).

[4]  Harith Alani,et al.  Geographical Information Retrieval with Ontologies of Place , 2001, COSIT.

[5]  Paolo Rosso,et al.  Inferring Geographical Ontologies from Multiple Resources for Geographical Information Retrieval , 2006, GIR.

[6]  Rossana M. de Castro Andrade,et al.  XMobile: A MB-UID environment for semi-automatic generation of adaptive applications for mobile devices , 2008, J. Syst. Softw..

[7]  Nigel Shadbolt,et al.  Image annotation with Photocopain , 2006 .

[8]  Jérôme Gensel,et al.  PhotoMap - Automatic Spatiotemporal Annotation for Mobile Photos , 2007, W2GIS.

[9]  Alan F. Smeaton,et al.  MediAssist: Using Content-Based Analysis and Context to Manage Personal Photo Collections , 2006, CIVR.

[10]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[11]  Alan F. Smeaton,et al.  Using text search for personal photo collections with the MediAssist system , 2007, SAC '07.

[12]  Chen Zhang,et al.  An empirical investigation of user term feedback in text-based targeted image search , 2007, TOIS.

[13]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track Report , 1999, TREC.

[14]  Pablo Castells,et al.  An Adaptation of the Vector-Space Model for Ontology-Based Information Retrieval , 2007, IEEE Transactions on Knowledge and Data Engineering.

[15]  Peng-Yuan Liu,et al.  Application-Oriented Comparison and Evaluation of Six Semantic Similarity Measures Based on Wordnet , 2006, 2006 International Conference on Machine Learning and Cybernetics.

[16]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[17]  Mor Naaman,et al.  Context data in geo-referenced digital photo collections , 2004, MULTIMEDIA '04.

[18]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[19]  Mor Naaman,et al.  Why we tag: motivations for annotation in mobile and online media , 2007, CHI.

[20]  Salima Benbernou,et al.  Query Rewriting for Semantic Multimedia Data Retrieval , 2008, Advances of Computational Intelligence in Industrial Systems.