Automatic Multimedia Knowledge Discovery, Summarization and Evaluation

This paper presents novel methods for automatically discovering, summarizing and evaluating multimedia knowledge from annotated images in the form of images clusters, word senses and relationships among them, among others. These are essential for applications to intelligently, efficiently and coherently deal with multimedia. The proposed methods include automatic techniques (1) for constructing perceptual knowledge by clustering the images based on visual and text feature descriptors, and discovering similarity and statistical relationships between the clusters; (2) for constructing semantic knowledge by disambiguating the senses of words in the annotations using WordNet and the images clusters, and finding semantic relations between the senses in WordNet; (3) for reducing the size of multimedia knowledge by clustering similar concepts together; and (4) for evaluating the quality of multimedia knowledge using information and graph theory notions. Experiments show the potential of integrating the analysis of images and annotations for improving the performance of the image clustering and the wordsense disambiguation, the importance of good concept distance measures for knowledge summarization, and the usefulness of automatic measures for evaluating knowledge quality.

[1]  Chris Brew,et al.  Using SGML as a Basis for Data-Intensive NLP , 1997, ANLP.

[2]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[3]  David A. Forsyth,et al.  Matching Words and Pictures , 2003, J. Mach. Learn. Res..

[4]  Graeme Hirst,et al.  Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures , 2004 .

[5]  Shih-Fu Chang,et al.  Video object model and segmentation for content-based video indexing , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[6]  David W. Conrath,et al.  Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy , 1997, ROCLING/IJCLCLP.

[7]  David Yarowsky,et al.  Unsupervised Word Sense Disambiguation Rivaling Supervised Methods , 1995, ACL.

[8]  Shih-Fu Chang,et al.  Perceptual knowledge construction from annotated image collections , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[9]  Shih-Fu Chang,et al.  Semantic knowledge construction from annotated image collections , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[10]  B. S. Manjunath,et al.  A Texture Thesaurus for Browsing Large Aerial Photographs , 1998, J. Am. Soc. Inf. Sci..

[11]  Shih-Fu Chang,et al.  Multimedia Knowledge Integration, Summarization And Evaluation , 2002, MDM/KDD.

[12]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[13]  Rada Mihalcea,et al.  Automatic generation of a coarse grained WordNet , 2001, HTL 2001.

[14]  Michael Sussna,et al.  Word sense disambiguation for free-text indexing using a massive semantic network , 1993, CIKM '93.

[15]  Kevin Murphy,et al.  Bayes net toolbox for Matlab , 1999 .

[16]  Neil C. Rowe,et al.  Precise and Efficient Retrieval of Captioned Images: The MARIE Project , 1999, Libr. Trends.

[17]  Alan F. Smeaton,et al.  Using WordNet in a Knowledge-Based Approach to Information Retrieval , 1995 .

[18]  William I. Grosky,et al.  Negotiating the semantic gap: from feature maps to semantic landscapes , 2001, Pattern Recognit..

[19]  Robert Tansley.  The multimedia thesaurus : adding a semantic layer to multimedia information , 2000 .

[20]  Michael J. Muller,et al.  VISAR: a system for inference and navigation of hypertext , 1989, Hypertext.

[21]  Rosalind W. Picard.  Toward a Visual Thesaurus , 1995, MIRO.

[22]  Shih-Fu Chang,et al.  IMKA: a multimedia organization system combining perceptual and semantic knowledge , 2001, MULTIMEDIA '01.

[23]  Shih-Fu Chang,et al.  MediaNet: a multimedia information network for knowledge representation , 2000, SPIE Optics East.

[24]  A. Shapiro.  Monte Carlo Sampling Methods , 2003 .

[25]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[26]  David G. Stork,et al.  Pattern Classification , 1973 .

[27]  Ray A. Jarvis,et al.  Clustering Using a Similarity Measure Based on Shared Near Neighbors , 1973, IEEE Transactions on Computers.

[28]  Richard A. Harshman,et al.  Indexing by latent semantic indexing , 1990 .

[29]  David A. Forsyth,et al.  Clustering art , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[30]  Alicia Perez,et al.  Evaluation of Taxonomic Knowledge in Ontologies and Knowledge Bases , 1999 .

[31]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[32]  Susan T. Dumais,et al.  Improving the retrieval of information from external sources , 1991 .

[33]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .