Context-aware mobile image annotation for media search and sharing

Abstract In recent years, rapid advances in media technology including acquisition, processing and distribution have led to proliferation of many mobile applications. Amongst them, one of the emerging applications is mobile-based image annotation that uses camera phones to capture images with system-suggested tags before uploading them to the media sharing portals. This procedure can offer information to mobile users and also facilitate the retrieval and sharing of the image for Web users. However, context information that can be acquired from mobile devices is underutilized in many existing mobile image annotation systems. In this paper, we propose a new mobile image annotation system that utilizes content analysis, context analysis and their integration to annotate images acquired from mobile devices. Specifically, three types of context, location, user interaction and Web, are considered in the tagging processes. An image dataset of Nanyang Technological University (NTU) campus has been constructed, and a prototype mobile image tag suggestion system has been developed. The experimental results show that the proposed system performs well in both effectiveness and efficiency on NTU dataset, and shows good potential in domain-specific mobile image annotation for image sharing.

[1]  Joo-Hwee Lim,et al.  Outdoor place recognition using compact local descriptors and multiple queries with user verification , 2007, ACM Multimedia.

[2]  James Ze Wang,et al.  Real-Time Computerized Annotation of Pictures , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Zhiwen Yu,et al.  FEMA: A Fast Expectation Maximization Algorithm based on Grid and PCA , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[4]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[5]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  Zhen Li,et al.  A Comparative Study of Mobile-Based Landmark Recognition Techniques , 2010, IEEE Intelligent Systems.

[7]  Mor Naaman,et al.  From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates , 2003, OTM.

[8]  Cordelia Schmid,et al.  Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[9]  Latifur Khan,et al.  Image annotations by combining multiple evidence & wordNet , 2005, ACM Multimedia.

[10]  Edward Y. Chang,et al.  CBSA: content-based soft annotation for multimodal image retrieval using Bayes point machines , 2003, IEEE Trans. Circuits Syst. Video Technol..

[11]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[12]  Tianwen Zhang,et al.  Using Example-Based Machine Translation Method For Automatic Image Annotation , 2006, 2006 6th World Congress on Intelligent Control and Automation.

[13]  James Ze Wang,et al.  Toward bridging the annotation-retrieval gap in image search by a generative modeling approach , 2006, MM '06.

[14]  Liviu Iftode,et al.  Context-aware Battery Management for Mobile Phones , 2008, 2008 Sixth Annual IEEE International Conference on Pervasive Computing and Communications (PerCom).

[15]  Daniel S. Yeung,et al.  Localized generalization error based active learning for image annotation , 2008, 2008 IEEE International Conference on Systems, Man and Cybernetics.

[16]  Lucas Paletta,et al.  A Mobile Vision System for Urban Detection with Informative Local Descriptors , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[17]  Min Zhang,et al.  Automatic image annotation based-on model space , 2005, 2005 International Conference on Natural Language Processing and Knowledge Engineering.

[18]  Tieniu Tan,et al.  Content-based rotation-invariant image annotation , 1996 .

[19]  Hichem Frigui,et al.  Unsupervised Image Segmentation and Annotation for Content-Based Image Retrieval , 2006, 2006 IEEE International Conference on Fuzzy Systems.

[20]  Gustavo Carneiro,et al.  Supervised Learning of Semantic Classes for Image Annotation and Retrieval , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Changhu Wang,et al.  Content-Based Image Annotation Refinement , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Kari Sentz,et al.  Combination of Evidence in Dempster-Shafer Theory , 2002 .

[23]  Mor Naaman,et al.  Zonetag: Designing context-aware mobile media capture to increase participation , 2006 .

[24]  Edward Y. Chang,et al.  Extent: Inferring Image Metadata from Context and Content , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[25]  Xiangyang Gong,et al.  Context-Aware Image Annotation and Retrieval on Mobile Device , 2010, 2010 Second International Conference on Multimedia and Information Technology.

[26]  A. Smeaton,et al.  Combination of content analysis and context features for digital photograph retrieval. , 2005 .