Exploiting information extraction techniques for automatic semantic video indexing with an application to Turkish news videos

This paper targets at the problem of automatic semantic indexing of news videos by presenting a video annotation and retrieval system which is able to perform automatic semantic annotation of news video archives and provide access to the archives via these annotations. The presented system relies on the video texts as the information source and exploits several information extraction techniques on these texts to arrive at representative semantic information regarding the underlying videos. These techniques include named entity recognition, person entity extraction, coreference resolution, and semantic event extraction. Apart from the information extraction components, the proposed system also encompasses modules for news story segmentation, text extraction, and video retrieval along with a news video database to make it a full-fledged system to be employed in practical settings. The proposed system is a generic one employing a wide range of techniques to automate the semantic video indexing process and to bridge the semantic gap between what can be automatically extracted from videos and what people perceive as the video semantics. Based on the proposed system, a novel automatic semantic annotation and retrieval system is built for Turkish and evaluated on a broadcast news video collection, providing evidence for its feasibility and convenience for news videos with a satisfactory overall performance.

[1]  Alberto Messina,et al.  A generalised cross-modal clustering method applied to multimedia news semantic indexing and retrieval , 2009, WWW '09.

[2]  Michael G. Strintzis,et al.  A System for the Semantic Multimodal Analysis of News Audio-Visual Content , 2010, EURASIP J. Adv. Signal Process..

[3]  M. Esmel ElAlami,et al.  Supporting image retrieval framework with rule base system , 2011, Knowl. Based Syst..

[4]  Mohan S. Kankanhalli,et al.  Multimodal fusion for multimedia analysis: a survey , 2010, Multimedia Systems.

[5]  Adnan Yazici,et al.  A Fuzzy Conceptual Model for Multimedia Data with a Text-Based Automatic Annotation Scheme , 2009, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[6]  Tobias Bjerregaard,et al.  A survey of research and practices of Network-on-chip , 2006, CSUR.

[7]  Dayne Freitag,et al.  Machine Learning for Information Extraction in Informal Domains , 2000, Machine Learning.

[8]  Fazli Can,et al.  Information retrieval on Turkish texts , 2008 .

[9]  Peter Jackson,et al.  Natural language processing for online applications : text retrieval, extraction and categorization , 2002 .

[10]  Valentin Tablan,et al.  Web-assisted annotation, semantic indexing and search of television and radio news , 2005, WWW '05.

[11]  Changsheng Xu,et al.  Semantic Event Extraction from Basketball Games using Multi-Modal Analysis , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[12]  Marco Bertini,et al.  Semantic annotation and retrieval of video events using multimedia ontologies , 2007 .

[13]  Roberto Basili,et al.  RitroveRAI: A Web Application for Semantic Indexing and Hyperlinking of Multimedia News , 2005, SEMWEB.

[14]  Ivar Jacobson,et al.  Unified Modeling Language User Guide, The (2nd Edition) (Addison-Wesley Object Technology Series) , 2005 .

[15]  Somnath Sengupta,et al.  Semantic concept mining in cricket videos for automated highlight generation , 2009, Multimedia Tools and Applications.

[16]  A. Yazici,et al.  Identification of coreferential chains in video texts for semantic annotation of news videos , 2008, 2008 23rd International Symposium on Computer and Information Sciences.

[17]  Adnan Yazici,et al.  Named Entity Recognition Experiments on Turkish Texts , 2009, FQAS.

[18]  E. Dikici,et al.  Sliding text recognition in broadcast news , 2008, 2008 IEEE 16th Signal Processing, Communication and Applications Conference.

[19]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[20]  Douglas E. Appelt,et al.  Introduction to Information Extraction Technology , 1999, IJCAI 1999.

[21]  Ivar Jacobson,et al.  The Unified Modeling Language User Guide , 1998, J. Database Manag..

[22]  John R. Smith,et al.  Semantic Indexing of Multimedia Content Using Visual, Audio, and Text Cues , 2003, EURASIP J. Adv. Signal Process..

[23]  Diane J. Cook,et al.  Automatic Video Classification: A Survey of the Literature , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[24]  Yafei Zhang,et al.  Dynamic Adaboost learning with feature selection based on parallel genetic algorithm for image annotation , 2010, Knowl. Based Syst..

[25]  Adnan Yazici,et al.  A Hybrid Named Entity Recognizer for Turkish with Applications to Different Text Genres , 2010, ISCIS.

[26]  Satoshi Sekine,et al.  A survey of named entity recognition and classification , 2007 .

[27]  Marcel Worring,et al.  Multimodal Video Indexing : A Review of the State-ofthe-art , 2001 .

[28]  Yi-Ping Phoebe Chen,et al.  Using object and trajectory analysis to facilitate indexing and retrieval of video , 2006, Knowl. Based Syst..

[29]  Changsheng Xu,et al.  Using Webcast Text for Semantic Event Detection in Broadcast Sports Video , 2008, IEEE Transactions on Multimedia.

[30]  Kalina Bontcheva,et al.  Multimedia indexing through multi-source and multi-language information extraction: the MUMIS project , 2004, Data Knowl. Eng..

[31]  Alicia Ageno,et al.  Adaptive information extraction , 2006, CSUR.

[32]  Antonio Valdovinos,et al.  Efficient Feedforward Linearization Technique Using Genetic Algorithms for OFDM Systems , 2010, EURASIP J. Adv. Signal Process..

[33]  Orkunt Sabuncu,et al.  Event Extraction from Turkish Football Web-casting Texts Using Hand-crafted Templates , 2009, 2009 IEEE International Conference on Semantic Computing.

[34]  Alan Hanjalic,et al.  Adaptive extraction of highlights from a sport video based on excitement modeling , 2005, IEEE Transactions on Multimedia.

[35]  Ruslan Mitkov,et al.  The Oxford handbook of computational linguistics , 2003 .

[36]  Rong Yan,et al.  A review of text and image retrieval approaches for broadcast news video , 2007, Information Retrieval.