Perceptual Narratives of Space and Motion for Semantic Interpretation of Visual Data

We propose a commonsense theory of space and motion for the high-level semantic interpretation of dynamic scenes. The theory provides primitives for commonsense representation and reasoning with qualitative spatial relations, depth profiles, and spatio-temporal change; these may be combined with probabilistic methods for modelling and hypothesising event and object relations. The proposed framework has been implemented as a general activity abstraction and reasoning engine, which we demonstrate by generating declaratively grounded visuo-spatial narratives of perceptual input from vision and depth sensors for a benchmark scenario.

[1]  Frank van Harmelen,et al.  Handbook of Knowledge Representation , 2008, Handbook of Knowledge Representation.

[2]  Philippe Muller,et al.  A Qualitative Theory of Motion Based on Spatio-Temporal Primitives , 1998, KR.

[3]  Anthony G. Cohn,et al.  Learning Functional Object-Categories from a Relational Spatio-Temporal Representation , 2008, ECAI.

[4]  Paulo E. Santos,et al.  Reasoning about Depth and Motion from an Observer's Viewpoint , 2007, Spatial Cogn. Comput..

[5]  Anthony G. Cohn,et al.  Building semantic scene models from unconstrained video , 2012, Comput. Vis. Image Underst..

[6]  Anthony G. Cohn,et al.  Constructing qualitative event models automatically from video input , 2000, Image Vis. Comput..

[7]  Gérard Ligozat.  Qualitative Spatial and Temporal Reasoning: Ligozat/Qualitative Spatial and Temporal Reasoning , 2013 .

[8]  Carl P. L. Schultz,et al.  Cognitive Interpretation of Everyday Activities - Toward Perceptual Narrative Based Visuo-Spatial Scene Interpretation , 2013, CMN.

[9]  Liang Wang,et al.  Semantic Understanding of Human Behaviors in Image Sequences: From video-surveillance to video-hermeneutics , 2012, Comput. Vis. Image Underst..

[10]  Ernest Davis.  Qualitative reasoning and spatio-temporal continuity , 2012 .

[11]  Mehul Bhatt,et al.  Reasoning about Space, Actions, and Change: A Paradigm for Applications of Spatial Reasoning , 2014 .

[12]  Lars Kulik.  Qualitative Spatial Change , 2002, Künstliche Intell..

[13]  Ehud Rivlin,et al.  Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[14]  Anthony G. Cohn,et al.  Event Model Learning from Complex Videos using ILP , 2010, ECAI.

[15]  Christian Freksa,et al.  Qualitative spatial reasoning , 1990, Forschungsberichte, TU Munich.

[16]  Travé-Massuyès.  Conceptual Neighborhood and its role in temporal and spatial reasoning , 1991 .

[17]  Hans-Hellmut Nagel,et al.  Cognitive Vision Systems, Sampling the Spectrum of Approaches [based on a Dagstuhl seminar] , 2006, Cognitive Vision Systems.

[18]  Antony Galton,et al.  Towards a Qualitative Theory of Movement , 1995, COSIT.

[19]  Monique Thonnat,et al.  Event Recognition System for Older People Monitoring Using an RGB-D Camera , 2013 .

[20]  David Vernon.  The Space of Cognitive Vision , 2006, Cognitive Vision Systems.

[21]  Sven J. Dickinson,et al.  A Research Roadmap of Cognitive Vision , 2005 .

[22]  Jiebo Luo,et al.  A Markov logic framework for recognizing complex events from multimodal data , 2013, ICMI '13.

[23]  A. U. Frank,et al.  Qualitative Spatial Reasoning , 2008, Encyclopedia of GIS.

[24]  Anthony G. Cohn,et al.  Qualitative Spatial Representation and Reasoning with the Region Connection Calculus , 1997, GeoInformatica.

[25]  P. Santos,et al.  Logic-based interpretation of geometrically observable changes occurring in dynamic scenes , 2009, Applied Intelligence.

[26]  Anthony G. Cohn,et al.  Unsupervised Learning of Event Classes from Video , 2010, AAAI.

[27]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[28]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[29]  Anthony G. Cohn,et al.  Qualitative Spatial Representation and Reasoning: An Overview , 2001, Fundam. Informaticae.

[30]  Carl P. L. Schultz,et al.  CLP(QS): A Declarative Spatial Reasoning Framework , 2011, COSIT.

[31]  Larry S. Davis,et al.  Event Modeling and Recognition Using Markov Logic Networks , 2008, ECCV.

[32]  Anthony G. Cohn,et al.  Abducing Qualitative Spatio-Temporal Histories from Partial Observations , 2002, KR.

[33]  Hans W. Guesgen,et al.  Qualitative Spatial and Temporal Reasoning: Emerging Applications, Trends, and Directions , 2011, Spatial Cogn. Comput..

[34]  Mark Witkowski,et al.  From Images to Bodies: Modelling and Exploiting Spatial Occlusion and Motion Parallax , 2001, IJCAI.

[35]  Anthony G. Cohn,et al.  Cognitive Vision: Integrating Symbolic Qualitative Representations with Computer Vision , 2006, Cognitive Vision Systems.

[36]  Antony Galton,et al.  Towards an Integrated Logic of Space, Time and Motion , 1993, IJCAI.

[37]  Larry S. Davis,et al.  Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Larry S. Davis,et al.  Multi-agent event recognition in structured scenarios , 2011, CVPR 2011.

[39]  David Vernon,et al.  Cognitive Vision: the Case for Embodied Perception , 2005 .

[40]  Patrick Koopmann,et al.  Ontology-Based Realtime Activity Monitoring Using Beam Search , 2011, ICVS.

[41]  Mehul Bhatt,et al.  Rotunde - A Smart Meeting Cinematography Initiative - Tools, Datasets, and Benchmarks for Cognitive Interpretation and Control , 2013, AAAI Workshop: Space, Time, and Ambient Intelligence.