Real-time virtual humans

The last few years have seen great maturation in the computation speed and control methods needed to portray 3D virtual humans that are suitable for real interactive applications. We first describe the state of the art, then focus on the particular approach taken at the University of Pennsylvania with the Jack system. Various aspects of real-time virtual humans are considered, such as their appearance and motion, interactive control, autonomous actions, gestures, attention, locomotion and multiple individuals. The underlying architecture consists of a sense-control-act structure that permits reactive behaviors to be locally adaptive to the environment, and a PaT-Net (parallel transition network) parallel-finite-state machine controller that can be used to drive virtual humans through complex tasks. We then argue for a deep connection between language and animation and describe current efforts in linking them through two systems: the Jack Presenter and the JackMOO extension to lambdaMOO. Finally, we outline a parameterized action representation for mediating between language instructions and animated actions.

[1]  Barry D. Reich,et al.  An architecture for behavioral locomotion , 1997 .

[2]  Norman I. Badler,et al.  Smart avatars in JackMOO , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[3]  Welton MacDonald Becket.  Reinforcement learning of reactive navigation for computer animation of simulated agents , 1997 .

[4]  Jugal K. Kalita,et al.  An Informal Semantic Analysis of Motion Verbs Based on Physical Primitives , 1997, Comput. Intell..

[5]  David Zeltzer,et al.  Task-level graphical simulation: abstraction, representation, and control , 1991 .

[6]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  N. Magnenat-Thalmann,et al.  Motion Control of Virtual Humans , 1998, IEEE Computer Graphics and Applications.

[8]  Joseph Bates,et al.  The role of emotion in believable agents , 1994, CACM.

[9]  Lance Williams,et al.  Motion signal processing , 1995, SIGGRAPH.

[10]  Jean-Claude Latombe,et al.  Planning motions with intentions , 1994, SIGGRAPH.

[11]  Norman I. Badler,et al.  Casualty Modeling for Real-Time Medical Training , 1996, Presence: Teleoperators & Virtual Environments.

[12]  Sharon A. Stansfield.  A Distributed Virtual Reality Simulation System for Situational Training , 1994, Presence: Teleoperators & Virtual Environments.

[13]  Daniel Thalmann,et al.  Simulation of object and human skin formations in a grasping task , 1989, SIGGRAPH.

[14]  Ioannis A. Kakadiaris,et al.  Building Anthropometry-Based Virtual Human Models , 1994 .

[15]  N.I. Badler,et al.  Avatars a/spl grave/ la Snow Crash , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[16]  Ioannis A. Kakadiaris,et al.  Model-based estimation of 3D human motion with occlusion based on active multi-viewpoint selection , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Norman I. Badler,et al.  Simulating humans: computer graphics animation and control , 1993 .

[18]  Daniel Thalmann,et al.  Dressing animated synthetic actors with complex deformable clothes , 1992, SIGGRAPH.

[19]  Christopher W. Geib,et al.  Planning for animation , 1996 .

[20]  Rodney A. Brooks,et al.  A robot that walks; emergent behaviors from a carefully evolved network , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[21]  Norman I. Badler,et al.  Kinematic control of human postures for task simulation , 1996 .

[22]  Andrew P. Witkin,et al.  Large steps in cloth simulation , 1998, SIGGRAPH.

[23]  Bonnie Webber,et al.  Animation from instructions , 1991 .

[24]  Alex Pentland,et al.  The ALIVE system: full-body interaction with autonomous agents , 1995, Proceedings Computer Animation'95.

[25]  Norman I. Badler,et al.  Digital Representations of Human Movement , 1979, CSUR.

[26]  Norman I. Badler,et al.  Animating expressivity through effort elements , 1999, SIGGRAPH '99.

[27]  Norman I. Badler,et al.  A Virtual Human Presenter , 1997 .

[28]  Mark Steedman,et al.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents , 1994, SIGGRAPH.

[29]  A. B. Loyall,et al.  Integrating Reactivity, Goals, and Emotion in a Broad Agent , 1992 .

[30]  Michael F. Cohen,et al.  Efficient generation of motion transitions using spacetime constraints , 1996, SIGGRAPH.

[31]  Norman I. Badler,et al.  Gesticulation behaviors for virtual humans , 1998, Proceedings Pacific Graphics '98. Sixth Pacific Conference on Computer Graphics and Applications (Cat. No.98EX208).

[32]  David C. Brogan,et al.  Animating human athletics , 1995, SIGGRAPH.

[33]  Norman I. Badler,et al.  Real-Time Inverse Kinematics of the Human Arm , 1996, Presence: Teleoperators & Virtual Environments.

[34]  Norman I. Badler,et al.  Instructions, Intentions and Expectations , 1995, Artif. Intell..

[35]  Ken Perlin,et al.  Improv: a system for scripting interactive actors in virtual worlds , 1996, SIGGRAPH.

[36]  Norman I. Badler,et al.  User-controlled physics-based animation for articulated figures , 1996, Proceedings Computer Animation '96.

[37]  Dimitris N. Metaxas.  Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging , 1996 .

[38]  Thomas W. Sederberg,et al.  Free-form deformation of solid geometric models , 1986, SIGGRAPH.

[39]  Ken Perlin,et al.  Real Time Responsive Animation with Personality , 1995, IEEE Trans. Vis. Comput. Graph..

[40]  Norman I. Badler,et al.  Terrain reasoning for human locomotion , 1994, Proceedings of Computer Animation '94.

[41]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[42]  Daniel Thalmann,et al.  Computer Animation , 1990, Computer Science Workbench.

[43]  Norman I. Badler,et al.  Making Them Move: Mechanics, Control & Animation of Articulated Figures , 1990 .

[44]  Christopher W. Geib,et al.  Planning and Terrain Reasoning , 1995 .

[45]  Norman I. Badler,et al.  Real-Time Control of a Virtual Human Using Minimal Sensors , 1993, Presence: Teleoperators & Virtual Environments.

[46]  Martha Palmer,et al.  Final Report to Air Force HRGA Regarding Feasibility of Natural Language Text Generation from Task Networks for Use in Automatic Generation of Technical Order from DEPTH Simulations , 1997 .

[47]  Norman I. Badler,et al.  Task-Level Object Grasping for Simulated Agents , 1996, Presence: Teleoperators & Virtual Environments.

[48]  Norman I. Badler,et al.  Decision networks for integrating the behaviors of virtual agents and avatars , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[49]  Daniel Thalmann,et al.  Computer Animation for Virtual Humans , 1998, IEEE Computer Graphics and Applications.

[50]  Anthony A. Maciejewski,et al.  Computational modeling for the computer animation of legged figures , 1985, SIGGRAPH.

[51]  Norman I. Badler,et al.  Avatars á la Snow Crash , 1998, CA.

[52]  David Salesin,et al.  Comic Chat , 1996, SIGGRAPH.

[53]  W. Lewis Johnson,et al.  Steve: an animated pedagogical agent for procedural training in virtual environments , 1997, SGAR.

[54]  Bond-Jay Ting.  Real-time human model design , 1998 .

[55]  Norman I. Badler,et al.  Inverse kinematics positioning using nonlinear programming for highly articulated figures , 1994, TOGS.

[56]  Hyeongseok Ko,et al.  Insertion of an articulated human into a networked virtual environment , 1994, Fifth Annual Conference on AI, and Planning in High Autonomy Systems.

[57]  Jane Wilhelms,et al.  Anatomically based modeling , 1997, SIGGRAPH.

[58]  Norman I. Badler,et al.  Animating human locomotion with inverse dynamics , 1996, IEEE Computer Graphics and Applications.

[59]  Libby Levison,et al.  Connecting planning and acting via object-specific reasoning , 1996 .

[60]  Sue Ki Wilcox.  Web Developer.com Guide to 3D Avatars , 1998 .

[61]  Kristinn R. Thórisson,et al.  Real-time decision making in multimodal face-to-face communication , 1998, AGENTS '98.