Automatic Semantic Annotation of Real-World Web Images

As the number of Web images is increasing at a rapid rate, searching them semantically presents a significant challenge. Many raw images are constantly uploaded with little meaningful direct annotations of semantic content, limiting their search and discovery. In this paper, we present a semantic annotation technique based on the use of image parametric dimensions and metadata. Using decision trees and rule induction, we develop a rule-based approach to formulate explicit annotations for images fully automatically, so that by the use of our method, semantic query such as "sunset by the sea in autumn in New York" can be answered and indexed purely by machine. Our system is evaluated quantitatively using more than 100,000 Web images. Experimental results indicate that this approach is able to deliver highly competent performance, attaining good recall and precision rates of sometimes over 80%. This approach enables a new degree of semantic richness to be automatically associated with images which previously can only be performed manually.

[1]  Xiaoping Chen,et al.  Ontology Based Object Categorization for Robots , 2005, PAKM.

[2]  Masashi Morimoto,et al.  Visual pattern discovery using web images , 2006, MIR '06.

[3]  Melanie Hilario,et al.  Distilling classification models from cross validation runs: an application to mass spectrometry , 2004, 16th IEEE International Conference on Tools with Artificial Intelligence.

[4]  Henry Lieberman,et al.  Beating Common Sense into Interactive Applications , 2004, AI Mag..

[5]  Rachid Deriche,et al.  A Review of Statistical Approaches to Level Set Segmentation: Integrating Color, Texture, Motion and Shape , 2007, International Journal of Computer Vision.

[6]  George Kollios,et al.  BoostMap: A method for efficient approximate similarity rankings , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[7]  Alessandro Perina,et al.  Natural scenes categorization by hierarchical extraction of typicality patterns , 2007, 14th International Conference on Image Analysis and Processing (ICIAP 2007).

[8]  Jürgen Gausemeier,et al.  Development of a real time image based object recognition method for mobile AR-devices , 2003, AFRIGRAPH '03.

[9]  Kobus Barnard,et al.  Exploiting Text and Image Feature Co-occurrence Statistics in Large Datasets , 2003 .

[10]  Junyu Dong,et al.  Combining Color, Texture and Region with Objects of User's Interest for Content-Based Image Retrieval , 2007, Eighth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD 2007).

[11]  Clement H. C. Leung,et al.  Structured natural-language descriptions for semantic content retrieval of visual materials , 2001, J. Assoc. Inf. Sci. Technol..

[12]  James Ze Wang,et al.  Real-Time Computerized Annotation of Pictures , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Heinrich H. Bülthoff,et al.  Categorization of natural scenes: Local versus global information and the role of color , 2007, TAP.

[14]  Adam Williams,et al.  Content-based image retrieval using joint correlograms , 2007, Multimedia Tools and Applications.

[15]  Michael I. Jordan,et al.  Modeling annotated data , 2003, SIGIR.

[16]  David A. Forsyth,et al.  Translating Images to Words for Recognizing Objects in Large Image and Video Collections , 2006, Toward Category-Level Object Recognition.

[17]  Bertrand Le Saux,et al.  Image recognition for digital libraries , 2004, MIR '04.

[18]  Tat-Seng Chua,et al.  A bootstrapping framework for annotating and retrieving WWW images , 2004, MULTIMEDIA '04.

[19]  Nuno Vasconcelos,et al.  From Pixels to Semantic Spaces: Advances in Content-Based Image Retrieval , 2007, Computer.

[20]  June-Suh Cho,et al.  Contour-based partial object recognition using symmetry in image databases , 2005, SAC '05.

[21]  Swarup Medasani,et al.  Content-based image retrieval based on a fuzzy approach , 2004, IEEE Transactions on Knowledge and Data Engineering.

[22]  Robin Lenman,et al.  The Oxford companion to the photograph , 2005 .

[23]  James Ze Wang,et al.  Content-based image retrieval: approaches and trends of the new age , 2005, MIR '05.

[24]  Thomas Vetter,et al.  Navigating in a Shape Space of Registered Models , 2007, IEEE Transactions on Visualization and Computer Graphics.

[25]  Torsten Rohlfing,et al.  Performance-based classifier combination in atlas-based image segmentation using expectation-maximization parameter estimation , 2004, IEEE Transactions on Medical Imaging.

[26]  John Tait,et al.  CLAIRE: A modular support vector image indexing and classification system , 2006, TOIS.

[27]  Yixin Chen,et al.  Content-based image retrieval by clustering , 2003, MIR '03.

[28]  C.H.C. Leung,et al.  Content-based image indexing and retrieval with XML representations , 2004, Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing, 2004..

[29]  Hongjin Huang,et al.  Robust Model Selection Using Cross Validation: A Simple Iterative Technique for Developing Robust Gene Signatures in Biomedical Genomics Applications , 2006, 2006 5th International Conference on Machine Learning and Applications (ICMLA'06).

[30]  Nicu Sebe,et al.  Context-Based Object-Class Recognition and Retrieval by Generalized Correlograms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  James Ze Wang,et al.  Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Djemel Ziou,et al.  Image Retrieval from the World Wide Web: Issues, Techniques, and Systems , 2004, CSUR.

[33]  Clement H. C. Leung,et al.  Implicit concept-based image indexing and retrieval , 2004, 10th International Multimedia Modelling Conference, 2004. Proceedings..

[34]  Joachim M. Buhmann,et al.  Robust Image Segmentation Using Resampling and Shape Constraints , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Rong Yan,et al.  Semantic concept-based query expansion and re-ranking for multimedia retrieval , 2007, ACM Multimedia.

[36]  Roberto Cipolla,et al.  Semantic Photo Synthesis , 2006, Comput. Graph. Forum.

[37]  Robert M. Gray,et al.  Histogram-based image retrieval using Gauss mixture vector quantization , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[38]  David A. Forsyth,et al.  Matching Words and Pictures , 2003, J. Mach. Learn. Res..

[39]  Paul Over,et al.  Multimedia retrieval benchmarks , 2004, IEEE MultiMedia.

[40]  Atilla Elçi,et al.  Semantic Annotation of Images , 2008, SWWS.

[41]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Tsuhan Chen,et al.  Content-Free Image Retrieval using Bayesian Product Rule , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[43]  Salvatore Ruggieri,et al.  Efficient C4.5 , 2002, IEEE Trans. Knowl. Data Eng..

[44]  Bo Thiesson,et al.  Image and Video Segmentation by Anisotropic Kernel Mean Shift , 2004, ECCV.

[45]  Dai Ran,et al.  A Sufficient and Necessary Condition for the Absolute Consistency of XML DTDs , 2007, Eighth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD 2007).