Animation from instructions

We believe that computer animation in the form of narrated animated simulations can provide an engaging, effective and flexible medium for instructing agents in the performance of tasks. However, we argue that the only way to achieve the kind of flexibility needed to instruct agents of varying capabilities to perform tasks with varying demands in work places of varying layout is to drive both animation and narration from a common representation that embodies the same conceptualization of tasks and actions as Natural Language itself. To this end, we are exploring the use of Natural Language instructions to drive animated simulations. In this paper, we discuss the relationship between instructions and behavior that underlie our work and the overall structure of our system. We then describe in some what more detail three aspects of the system the representation used by the Simulator, the operation of the Simulator and the Motion Generators used in the system. Comments University of Pennsylvania Department of Computer and Information Science Technical Report No. MSCIS-90-17. This technical report is available at ScholarlyCommons: http://repository.upenn.edu/cis_reports/533 Animation From Instructions MS-CIS-90-17 GRAPHICS LAB 33 LINC LAB 165 Norman I. Badler Bonnie L. Webber Jeffrey Esakov Jugal Kalita Department of Computer and Information Science School of Engineering and Applied Science University of Pennsylvania Philadelphia, PA 19104-6389

[1]  Michael E. Bratman.  Taking Plans Seriously , 1983 .

[2]  N. Badler,et al.  The Simulation of Human Movement by Computer , 1978 .

[3]  J. O'Rourke,et al.  Model-based image analysis of human motion using constraint propagation , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Andrew P. Witkin,et al.  Energy constraints on parameterized models , 1987, SIGGRAPH.

[5]  Norman I. Badler,et al.  Making Them Move: Mechanics, Control & Animation of Articulated Figures , 1990 .

[6]  Daniel Thalmann,et al.  The Direction of Synthetic Actors in the Film Rendez-Vous a Montreal , 1987, IEEE Computer Graphics and Applications.

[7]  Norman I. Badler,et al.  Interactive real-time articulated figure manipulation using multiple kinematic constraints , 1990, I3D '90.

[8]  W ReynoldsCraig.  Flocks, herds and schools: A distributed behavioral model , 1987 .

[9]  P. Morasso,et al.  Understanding Human Movement , 1986 .

[10]  Norman I. Badler,et al.  An architecture for the simulation of human movement , 1978, ACM '78.

[11]  N. Badler,et al.  Real Time Inverse Kinematics with Joint Limits and Spatial Constraints , 1989 .

[12]  Noah Webster,et al.  Webster's new collegiate dictionary , 1936 .

[13]  Ernest M. Otani.  Software Tools for Dynamic and Kinematic Modeling of Human Emotion , 1989 .

[14]  Jay Earley,et al.  An efficient context-free parsing algorithm , 1970, Commun. ACM.

[15]  Josh D. Tenenberg.  Inheritance in Automated Planning , 1989, KR.

[16]  Annette Herskovits,et al.  Language and spatial cognition , 1986 .

[17]  Paul Anthony Fishwick.  Hierarchical reasoning: simulating complex processes over multiple levels of abstraction , 1986 .

[18]  Thomas W. Calvert,et al.  Goal-directed, dynamic animation of human walking , 1989, SIGGRAPH.

[19]  Norman I. Badler,et al.  Strength guided motion , 1990, SIGGRAPH.

[20]  Robin F. Karlin.  Defining the Semantics of Verbal Modifiers in the Domain of Cooking Tasks , 1988, ACL.

[21]  Jane Wilhelms,et al.  Collision Detection and Response for Computer Animation , 1988, SIGGRAPH.

[22]  Jeffrey Esakov,et al.  An Investigation of Language Input and Performance Timing for Task Animation , 1988 .

[23]  Norman I. Badler,et al.  Techniques for Goal-Directed Motion , 1981 .

[24]  J. O'Rourke,et al.  A spherical representation of a human body for visualizing movement , 1979, Proceedings of the IEEE.

[25]  Stephen Michael Platt,et al.  A structural model of the human face (graphics, animation, object representation) , 1985 .

[26]  Anne Abeillé,et al.  Parsing Idioms in Lexicalized TAGs , 1989, EACL.

[27]  Norman I. Badler,et al.  Temporal scene analysis: conceptual descriptions of object movements. , 1975 .

[28]  Paul A. Fishwick,et al.  The role of process abstraction in simulation , 1988, IEEE Trans. Syst. Man Cybern..

[29]  Norman I. Badler,et al.  Parametric keyframe interpolation incorporating kinetic adjustment and phrasing control , 1985, SIGGRAPH.

[30]  Michael F. Cohen,et al.  Controlling dynamic simulation with kinematic constraints , 1987, SIGGRAPH.

[31]  Diana T. Dadamo.  Effective Control of Human Motion Animation , 1988 .

[32]  Warren Robinett,et al.  Virtual environment display system , 1987, I3D '86.

[33]  Lucy Suchman.  Plans and situated actions: the problem of human-machine communication , 1987 .

[34]  David Chapman,et al.  What are plans for? , 1990, Robotics Auton. Syst..

[35]  John E. Howland,et al.  Computer graphics , 1990, IEEE Potentials.

[36]  Bonnie L. Webber,et al.  The Interpretation of Tense in Discourse , 1987, ACL.

[37]  Jeffrey Scott Gangel,et al.  A Motion Verb Interface to a Task Animation System , 1985 .

[38]  A. Goldman.  Theory of Human Action , 1970 .

[39]  David Zeltzer,et al.  Towards an integrated view of 3-D computer animation , 1986 .

[40]  Andrew P. Witkin,et al.  Spacetime constraints , 1988, SIGGRAPH.

[41]  Robert Lake,et al.  Near-Real-Time Control of Human Figure Models , 1987, IEEE Computer Graphics and Applications.

[42]  Jane Wilhelms,et al.  Using Dynamic Analysis for Realistic Animation of Articulated Bodies , 1987, IEEE Computer Graphics and Applications.

[43]  Mark Steedman,et al.  Grammar, interpretation, and processing from the lexicon , 1989 .

[44]  James U. Korein,et al.  A geometric investigation of reach , 1985 .

[45]  Norman I. Badler,et al.  Imperfection for realistic image synthesis , 1990, Comput. Animat. Virtual Worlds.

[46]  Bonnie L. Webber,et al.  Tense as Discourse Anaphor , 1988, CL.

[47]  Norman I. Badler,et al.  Articulated Figure Positioning by Multiple Constraints , 1987, IEEE Computer Graphics and Applications.

[48]  Bernard Meltzer,et al.  Analogical Representations of Naive Physics , 1989, Artif. Intell..

[49]  John E. Hopcroft,et al.  Simulation of physical systems from geometric models , 1987, IEEE J. Robotics Autom..

[50]  Martha E. Pollack,et al.  Inferring domain plans in question-answering , 1986 .

[51]  James K. Hahn,et al.  Realistic animation of rigid bodies , 1988, SIGGRAPH.

[52]  Zeltzer,et al.  Motor Control Techniques for Figure Animation , 1982, IEEE Computer Graphics and Applications.

[53]  Rebecca J. Passonneau,et al.  A Computational Model of the Semantics of Tense and Aspect , 1988, CL.

[54]  Arun N. Netravali,et al.  Motion interpolation by optimal control , 1988, SIGGRAPH.

[55]  Steven K. Feiner,et al.  Apex: An Experiment in the Automated Creation of Pictorial Explanations , 1985, IEEE Computer Graphics and Applications.

[56]  John K. Tsotsos,et al.  Goal directed animation using English motion commands , 1986 .

[57]  Mark Steedman,et al.  A Lazy way to Chart-Parse with Categorial Grammars , 1987, ACL.

[58]  Gary G. Hendrix,et al.  Modeling Simultaneous Actions and Continuous Processes , 1989, Artif. Intell..

[59]  Norman I. Badler,et al.  Anthropometry for Computer Graphics Human Figures , 1989 .

[60]  Earl David Sacerdoti,et al.  A Structure for Plans and Behavior , 1977 .

[61]  Philip R. Cohen.  The Need for Referent Identification as a Planned Action , 1981, IJCAI.

[62]  T W Calvert,et al.  The interactive specification of human animation , 1986 .

[63]  Chris Mellish,et al.  Natural Language Generation from Plans , 1989, Comput. Linguistics.

[64]  Y. Takashima,et al.  Story driven animation , 1987, CHI '87.

[65]  Steven K. Feiner,et al.  Coordinating Text and Graphics in Explanation Generation , 1989, HLT.

[66]  Ronen Barzel,et al.  A modeling system based on dynamic constraints , 1988, SIGGRAPH.

[67]  Barbara Di Eugenio,et al.  Free Adjuncts in Natural Language Instructions , 1990, COLING.

[68]  Mark Green,et al.  Interactive animation: a language and system for procedural modeling and motion , 1988, IEEE Computer Graphics and Applications.

[69]  R. F. Simmons,et al.  Semantically Analyzing an English Subset for the Clowns Microworld , 1975, CL.

[70]  Kaufman,et al.  A New Color-Namiing System for Graphics Languages , 1982, IEEE Computer Graphics and Applications.

[71]  Robert Dale,et al.  Generating referring expressions in a domain of objects and processes (language representation) , 1988 .

[72]  Anthony A. Maciejewski,et al.  Computational modeling for the computer animation of legged figures , 1985, SIGGRAPH.

[73]  Mark Steedman,et al.  Temporal Ontology and Temporal Reference , 1988, CL.

[74]  J Wilhelms,et al.  Virya—a motion control editor for kinematic and dynamic animation , 1986 .

[75]  Norman I. Badler,et al.  JACK: a toolkit for manipulating articulated figures , 1988, UIST '88.

[76]  C. R.,et al.  On referring , 1950 .

[77]  William D. Marslen-Wilson,et al.  Lexical Representation and Process , 1991 .

[78]  Norman I. Badler,et al.  Special problems in human movement simulation , 1980, SIGGRAPH '80.

[79]  Norman I. Badler,et al.  Digital Representations of Human Movement , 1979, CSUR.

[80]  Z. Vendler.  Linguistics in Philosophy , 1967 .

[81]  F. Thomas,et al.  Disney Animation: The Illusion of Life , 1981 .

[82]  Norman I. Badler,et al.  Constraint-Based Temporal Planning , 1988 .

[83]  C. Sidner,et al.  Plans for Discourse , 1988 .