Playing games as a way to improve automatic image annotation

Image annotation is hard to do in an automatic way. In this paper, we propose a framework for image annotation that combines the benefits of three paradigms: automatic annotation, human intervention and entertainment activities. We also describe our proposal inside this framework, the ASAA (application for semi-automatic annotation) interface, a new computer game for image tagging. The application has a 3D game interface, and is supported by a game engine that uses a system for automatic image classification and gestural input to play the game. We present results of the performance of semantic models obtained with a training set enlarged by images annotated during the game activity as well as usability tests of the application.

[1]  Paul Over,et al.  TRECVID 2006 Overview , 2006, TRECVID.

[2]  Jürgen Scheible,et al.  Combining Web, Mobile Phones and Public Displays in Large-Scale: Manhattan Story Mashup , 2007, Pervasive.

[3]  Allan Kuchinsky,et al.  Requirements for photoware , 2002, CSCW '02.

[4]  Jianping Fan,et al.  Hierarchical classification for automatic image annotation , 2007, SIGIR.

[5]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[6]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[7]  Edward Y. Chang,et al.  Support vector machine active learning for image retrieval , 2001, MULTIMEDIA '01.

[8]  Bir Bhanu,et al.  Reinforcement learning for combining relevance feedback techniques , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[9]  Nicu Sebe,et al.  Content-based multimedia information retrieval: State of the art and challenges , 2006, TOMCCAP.

[10]  Y. Mori,et al.  Image-to-word transformation based on dividing and vector quantizing images with words , 1999 .

[11]  Alexander J. Smola,et al.  Advances in Large Margin Classifiers , 2000 .

[12]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[13]  Wei-Ying Ma,et al.  AnnoSearch: Image Auto-Annotation by Search , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[14]  Mary Czerwinski,et al.  Semi-Automatic Image Annotation , 2001, INTERACT.

[15]  Simon King,et al.  From context to content: leveraging context to infer media metadata , 2004, MULTIMEDIA '04.

[16]  Rong Yan,et al.  An efficient manual image annotation approach based on tagging and browsing , 2007, MS '07.

[17]  Qiang Yang,et al.  A unified framework for semantics and feature based relevance feedback in image retrieval systems , 2000, ACM Multimedia.

[18]  R. Manmatha,et al.  A Model for Learning the Semantics of Pictures , 2003, NIPS.

[19]  Daniel Gatica-Perez,et al.  On image auto-annotation with latent space models , 2003, ACM Multimedia.

[20]  Shih-Fu Chang,et al.  Semantic visual templates: linking visual features to semantics , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[21]  Milind R. Naphade,et al.  A probabilistic framework for semantic video indexing, filtering, and retrieval , 2001, IEEE Trans. Multim..

[22]  Stefan M. Rüger,et al.  Information-theoretic semantic multimedia indexing , 2007, CIVR '07.

[23]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  A. Abrantes,et al.  Sharing Personal Experiences while Navigating in Physical Spaces , 2007 .

[25]  Thomas S. Huang,et al.  Relevance feedback in image retrieval: A comprehensive review , 2003, Multimedia Systems.

[26]  David A. Forsyth,et al.  Learning the semantics of words and pictures , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[27]  Jiebo Luo,et al.  Pictures are not taken in a vacuum - an overview of exploiting context for semantic scene content understanding , 2006, IEEE Signal Processing Magazine.

[28]  James Ze Wang,et al.  Real-Time Computerized Annotation of Pictures , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Stefan M. Rüger,et al.  Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation , 2005, CIVR.

[30]  Nuno Correia,et al.  A gesture based game for image tagging , 2008, CHI Extended Abstracts.

[31]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.