Generation of augmented video sequences combining behavioral animation and multi‐object tracking

In this paper we present a novel approach to generate augmented video sequences in real‐time, involving interactions between virtual and real agents in real scenarios. On the one hand, real agent motion is estimated by means of a multi‐object tracking algorithm, which determines real objects' position over the scenario for each time step. On the other hand, virtual agents are provided with behavior models considering their interaction with the environment and with other agents. The resulting framework allows to generate video sequences involving behavior‐based virtual agents that react to real agent behavior and has applications in education, simulation, and in the game and movie industries. We show the performance of the proposed approach in an indoor and outdoor scenario simulating human and vehicle agents. Copyright © 2009 John Wiley & Sons, Ltd.

[1]  Hujun Bao,et al.  As‐consistent‐As‐possible compositing of virtual objects and video sequences , 2006, Comput. Animat. Virtual Worlds.

[2]  Charles E. Hughes,et al.  Mixed reality in education, entertainment, and training , 2005, IEEE Computer Graphics and Applications.

[3]  Bruce Blumberg,et al.  Multi-level direction of autonomous creatures for real-time virtual environments , 1995, SIGGRAPH.

[4]  Pau Baiget,et al.  Automatic generation of computer animated sequences based on human behaviour modelling , 2007 .

[5]  Ioannis Pitas,et al.  Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines , 2007, IEEE Transactions on Image Processing.

[6]  Jordi Gonzàlez,et al.  Background subtraction technique based on chromaticity and intensity patterns , 2008, 2008 19th International Conference on Pattern Recognition.

[7]  Daniel Thalmann,et al.  Interaction between real and virtual humans in augmented reality , 1997, Proceedings. Computer Animation '97 (Cat. No.97TB100120).

[8]  F. Xavier Roca,et al.  Understanding dynamic scenes based on human sequence evaluation , 2009, Image Vis. Comput..

[9]  José M. F. Moura,et al.  Capture and Representation of Human Walking in Live Video Sequences , 1999, IEEE Trans. Multim..

[10]  Daniel Thalmann,et al.  Behavioral Animation of Autonomous Virtual Agents Helped By Reinforcement , 2003 .

[11]  Bruce H. Thomas,et al.  An object-oriented software architecture for 3D mixed reality applications , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[12]  Dieter Schmalstieg,et al.  Augmented reality agents for user interface adaptation , 2008, Comput. Animat. Virtual Worlds.

[13]  Daniel Thalmann,et al.  Animating virtual actors in real environments , 1997, Multimedia Systems.

[14]  Erol Gelenbe,et al.  Simulating Autonomous Agents with augmented Reality , 2001, GAME-ON.

[15]  Ronen Basri,et al.  Actions as space-time shapes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[16]  Fadi Dornaika,et al.  Simultaneous Facial Action Tracking and Expression Recognition in the Presence of Head Motion , 2008, International Journal of Computer Vision.

[17]  Norman I. Badler,et al.  Animation control for real-time virtual humans , 1999, CACM.

[18]  Ken Perlin,et al.  Improv: a system for scripting interactive actors in virtual worlds , 1996, SIGGRAPH.

[19]  Hans-Hellmut Nagel,et al.  Behavioral Knowledge Representation for the Understanding and Creation of Video Sequences , 2003, KI.

[20]  George Papagiannakis,et al.  A survey of mobile and wireless technologies for augmented reality systems , 2008, Comput. Animat. Virtual Worlds.

[21]  Hans-Hellmut Nagel,et al.  Conceptual representations between video signals and natural language descriptions , 2008, Image Vis. Comput..

[22]  F BobickAaron,et al.  The Recognition of Human Movement Using Temporal Templates , 2001 .

[23]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[24]  Pau Baiget,et al.  Autonomous Virtual Agents for Performance Evaluation of Tracking Algorithms , 2008, AMDO.

[25]  George Papagiannakis,et al.  Mixing virtual and real scenes in the site of ancient Pompeii , 2005, Comput. Animat. Virtual Worlds.

[26]  Xavier Varona,et al.  Automatic Keyframing of Human Actions for Computer Animation , 2003, IbPRIA.

[27]  Matthijs Douze,et al.  Real‐time generation of augmented video sequences by background tracking , 2006, Comput. Animat. Virtual Worlds.

[28]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[29]  Daniel Thalmann,et al.  Autonomous Virtual Agents Learning a Cognitive Model and Evolving , 2005, IVA.

[30]  Ian D. Reid,et al.  Colour Invariant Head Pose Classification in Low Resolution Video , 2008, BMVC.

[31]  Daniel Rowe.  Towards Robust Multiple-Target Tracking in Unconstrained Human-Populated Environments , 2008 .

[32]  Hans-Hellmut Nagel,et al.  Incremental recognition of traffic situations from video image sequences , 2000, Image Vis. Comput..

[33]  Daniel Thalmann,et al.  Learnable behavioural model for autonomous virtual agents: low-level learning , 2006, AAMAS '06.

[34]  Alain Colmerauer,et al.  An introduction to Prolog III , 1989, CACM.