Semantic modeling and management of multimedia data

The objective of this research is two fold: (i) semantic modeling and management of video data; (ii) modeling of and management of multimedia documents. A semantic-based abstraction of video data for capturing the spatio-temporal concepts associated with various objects in an input image or in a sequence of video frames is proposed. This abstraction can manifest itself effectively in conceptualizing events and views in multimedia data as perceived by individual users. The goal is to provide an efficient mechanism for handling content-based queries, with the minimum amount of processing performed on raw data during query evaluation. Based on shared semantics, information can be integrated at various levels of abstraction leading to a multi-level indexing/searching mechanism. At the finest level of granularity, video data can be indexed based on simple contents such as the appearance of objects and faces. For management of information at higher levels of abstractions, an object-oriented paradigm is proposed which provides a comprehensive video database architecture. For document modeling and management, a Petri-Net-based model is proposed. This model takes into account the temporal, spatial, and content information of the component objects. The integration of this model with logical document structures and hypermedia environments is suggested. Algorithms for clustering multimedia documents based on user-specified concepts are presented. Consequently, a multimedia query calculus that is built upon a set of generalized n-ary spatio-temporal relations for multimedia data is proposed. The formal syntax and semantics of the query calculus are presented. This query calculus supports various functionalities such as declaration of multimedia data, specification of spatio-temporal logic, expression of spatio-temporal semantics for content-based retrieval of image/video data, composition of multimedia documents, and orchestration of presentations. The computability aspects of the proposed calculus are also discussed.