A Similarity-Based Approach for Audiovisual Document Classification Using Temporal Relation Analysis

We propose a novel approach for video classification that bases on the analysis of the temporal relationships between the basic events in audiovisual documents. Starting from basic segmentation results, we define a new representation method that is called Temporal Relation Matrix (TRM). Each document is then described by a set of TRMs, the analysis of which makes events of a higher level stand out. This representation has been first designed to analyze any audiovisual document in order to find events that may well characterize its content and its structure. The aim of this work is to use this representation to compute a similarity measure between two documents. Approaches for audiovisual documents classification are presented and discussed. Experimentations are done on a set of 242 video documents and the results show the efficiency of our proposals.

[1]  Ruud M. Bolle,et al.  Feature based indexing for media tracking , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[2]  Ambuj K. Singh,et al.  Variable length queries for time series data , 2001, Proceedings 17th International Conference on Data Engineering.

[3]  P. Joly,et al.  Audio Data Analysis using Parametric Representation of Temporal Relations , 2006, 2006 2nd International Conference on Information & Communication Technologies.

[4]  John R. Smith,et al.  Semi-automatic, data-driven construction of multimedia ontologies , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[5]  Henry A. Kautz,et al.  Constraint Propagation Algorithms for Temporal Reasoning , 1986, AAAI.

[6]  Simone Santini,et al.  Similarity Measures , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Diane J. Cook,et al.  Automatic Video Classification: A Survey of the Literature , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[8]  Sid-Ahmed Berrani,et al.  Content-Based Video Segment Reunification for TV Program Extraction , 2009, 2009 11th IEEE International Symposium on Multimedia.

[9]  Patrick Gros,et al.  AVSST: an Automatic Video Stream Structuring Tool , 2010 .

[10]  Christos Faloutsos,et al.  Efficient Similarity Search In Sequence Databases , 1993, FODO.

[11]  Angelo Montanari,et al.  Trends in temporal representation and reasoning , 1996, The Knowledge Engineering Review.

[12]  Olivier Buisson,et al.  Local Behaviours Labelling for Content Based Video Copy Detection , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[13]  Bernhard Nebel,et al.  Reasoning about temporal relations: a maximal tractable subclass of Allen's interval algebra , 1994, JACM.

[14]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[15]  Yueting Zhuang,et al.  Content-based video similarity model , 2000, ACM Multimedia.

[16]  Jonathan Foote,et al.  Media segmentation using self-similarity decomposition , 2003, IS&T/SPIE Electronic Imaging.

[17]  Richard J. Qian,et al.  Detecting semantic events in soccer games: towards a complete solution , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[18]  Peter van Beek,et al.  Exact and approximate reasoning about temporal relations 1 , 1990, Comput. Intell..

[19]  Sid-Ahmed Berrani,et al.  Automatic TV Broadcast Structuring , 2010, Int. J. Digit. Multim. Broadcast..

[20]  Eric Bruno,et al.  Prédiction Temporelle de Descripteurs Visuels pour la Mesure de Similarité entre Vidéos , 2003 .

[21]  Ron Shamir,et al.  Complexity and algorithms for reasoning about time: a graph-theoretic approach , 1993, JACM.

[22]  Piotr Indyk,et al.  Similarity Search in High Dimensions via Hashing , 1999, VLDB.

[23]  Rainer Lienhart,et al.  Reliable Transition Detection in Videos: A Survey and Practitioner's Guide , 2001, Int. J. Image Graph..

[24]  G. P. Bhattacharjee,et al.  Temporal representation and reasoning in artificial intelligence: A review , 2001 .

[25]  Bertrand Delezoide.  HIERARCHICAL FILM SEGMENTATION USING AUDIO AND VISU AL SIMILARITY , 2005 .

[26]  Philippe Joly,et al.  Temporal Relation Analysis in Audiovisual Documents for Complementary Descriptive Information , 2005, Adaptive Multimedia Retrieval.

[27]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[28]  Angelo Montanari,et al.  Temporal representation and reasoning in artificial intelligence: Issues and approaches , 2000, Annals of Mathematics and Artificial Intelligence.

[29]  Peter Jonsson,et al.  Twenty-One Large Tractable Subclasses of Allen's Algebra , 1997, Artif. Intell..

[30]  Zein Al Abidin Ibrahim,et al.  Caractérisation des structures audiovisuelles par analyse statistique des relations temporelles , 2007 .

[31]  Sanjeev R. Kulkarni,et al.  A framework for measuring video similarity and its application to video query by example , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[32]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[33]  Gérard Ligozat,et al.  On Generalized Interval Calculi , 1991, AAAI.

[34]  Anoop Gupta,et al.  Automatically extracting highlights for TV Baseball programs , 2000, ACM Multimedia.

[35]  Cormac Herley,et al.  Extracting repeats from media streams , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[36]  Philippe Joly,et al.  Improvement of a Person Labelling Method Using Extracted Knowledge on Costume , 2005, CAIP.

[37]  James F. Allen.  An Interval-Based Representation of Temporal Knowledge , 1981, IJCAI.

[38]  Li Zhao,et al.  Method Based On Temporal Constrain Shot Method Based On Temporal Constrain Shot Similarity , 2001, ICME.

[39]  Patrick Gros,et al.  Fast Structuring of Large Television Streams Using Program Guides , 2006, Adaptive Multimedia Retrieval.

[40]  Qi Tian,et al.  A unified framework for semantic shot classification in sports video , 2002, IEEE Transactions on Multimedia.

[41]  Lluís Vila,et al.  A Survey on Temporal Reasoning in Artificial Intelligence , 1994, AI Communications.

[42]  Henry A. Kautz,et al.  Integrating Metric and Qualitative Temporal Reasoning , 1991, AAAI.

[43]  Avideh Zakhor,et al.  Efficient video similarity measurement with video signature , 2002, Proceedings. International Conference on Image Processing.

[44]  Rina Dechter,et al.  Temporal Constraint Networks , 1989, Artif. Intell..

[45]  Henry A. Kautz,et al.  Constraint propagation algorithms for temporal reasoning: a revised report , 1989 .

[46]  Itay Meiri,et al.  Combining Qualitative and Quantitative Constraints in Temporal Reasoning , 1991, Artif. Intell..

[47]  Christos Faloutsos,et al.  Compressed-domain video indexing techniques using DCT and motion vector information in MPEG video , 1997, Electronic Imaging.

[48]  Yannis Avrithis,et al.  Broadcast news parsing using visual cues: a robust face detection approach , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[49]  Stefan Eickeler,et al.  Content-based video indexing of TV broadcast news using hidden Markov models , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[50]  Bernard Moulin,et al.  Conceptual-graph approach for the representation of temporal information in discourse , 1992, Knowl. Based Syst..

[51]  Siba Haidar,et al.  Mining for video production invariants to measure style similarity , 2006, Int. J. Intell. Syst..