Recognizing intentions, interactions, and causes of plan failures

Natural language systems for the description of image sequences have been developed (e.g. Neumann and Novak, 1986; Herzog et al., 1989). Even though these systems were used to verbalize the behaviour of human agents, they were limited in that they could only describe the purely visual, i.e. spatio-temporal, properties of the behaviour observed. For many applications of such systems (e.g. co-driver systems in traffic, expert systems in high-performance sports, tutorial systems that give “apprentices” instructions in construction tasks, etc.), it seems useful to extend their capabilities to cover a greater part of the performance of a human observer and thus make the system more helpful to the user. In particular, an interpretation process ought to be modelled that yields hypotheses about intentional entities from spatio-temporal information about agents. Its results should be verbalized in natural language. This article presents an integrated approach for the recognition and natural language description of plans, intentions, interactions between multiple agents, plan failures, and causes of plan failures. The system described takes observations from image sequences as input. This type of input poses specific problems for the recognition process. After moving objects have been extracted from the image sequences by a vision system and spatio-temporal entities (such as spatial relations and events) have been recognized by an event-recognition system, a focussing process selects interesting agents to be concentrated on in the plan-recognition process. The set of plan hypotheses can be reduced by a hypothesis-selection component. Plan recognition serves as the basis for intention recognition, interaction recognition, and plan-failure analysis. The recognized intentional entities are described in natural language. The system is designed to extend the range of capabilities of the system SOCCER, which verbalizes real-world image sequences of soccer games in natural language.

[1]  Eric Werner,et al.  Toward a Theory of Communication and Cooperation for Multiagent Planning , 1988, TARK.

[2]  James F. Allen,et al.  A Plan Recognition Model for Subdialogues in Conversations , 1987, Cogn. Sci..

[3]  N. S. Sridharan,et al.  The Plan Recognition Problem: An Intersection of Psychology and Artificial Intelligence , 1978, Artif. Intell..

[4]  Candace L. Sidner,et al.  Models of Plans to Support Communication: An Initial Report , 1990, AAAI.

[5]  Kurt Konolige,et al.  Ascribing Plans to Agents , 1989, IJCAI.

[6]  Bertram C. Bruce,et al.  Interacting plans , 1978, SGAR.

[7]  David J. Israel,et al.  Plans and resource‐bounded practical reasoning , 1988, Comput. Intell..

[8]  Keith S. Donnellan,et al.  Action, Emotion And Will , 1965 .

[9]  Jaegwon Kim.  Intention and Practical Inference , 1976 .

[10]  C. Raymond Perrault,et al.  Beyond Question-Answering. , 1981 .

[11]  Michael P. Georgeff,et al.  Communication and interaction in multi-agent planning , 1983, AAAI 1983.

[12]  Henry A. Kautz,et al.  Generalized Plan Recognition , 1986, AAAI.

[13]  C. Raymond Perrault,et al.  Analyzing Intention in Utterances , 1986, Artif. Intell..

[14]  A. Baier,et al.  Intention, Practical Knowledge and Representation , 1976 .

[15]  Douglas E. Appelt,et al.  Planning English Sentences , 1988, Cogn. Sci..

[16]  Hans-Hellmut Nagel,et al.  PROMETHEUS Session , 1990, ECAI.

[17]  G. Miller,et al.  Language and Perception , 1976 .

[18]  J. Carbonell.  The Counterplanning Process: A Model of Decision-Making in Adverse Situations , 1979 .

[19]  Sandra Carberry,et al.  Plan Recognition and Its Use in Understanding Dialog , 1989 .

[20]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[21]  Hector J. Levesque,et al.  On Acting Together , 1990, AAAI.

[22]  Robert V. London.  Student modeling to support multiple instructional approaches , 1992, User Modeling and User-Adapted Interaction.

[23]  Philip R. Cohen,et al.  Collective Intentions and Actions , 2003 .

[24]  Bernd Neumann,et al.  NOAS: Ein System zur natürlichsprachlichen Beschreibung zeitveränderlicher Szenen , 1986, Inform. Forsch. Entwickl..

[25]  Candace L. Sidner,et al.  Plan parsing for intended response recognition in discourse 1 , 1985, Comput. Intell..

[26]  Philip R. Cohen,et al.  Discourse Processing and Commonsense Plans , 2003 .

[27]  Drew McDermott,et al.  Introduction to artificial intelligence , 1986, Addison-Wesley series in computer science.

[28]  Gordon I. McCalla,et al.  A Computational Framework for Granularity and its Application to Educational Diagnosis , 1989, IJCAI.

[29]  M. Brady,et al.  Recognizing Intentions From Natural Language Utterances , 1983 .

[30]  Joachim Hertzberg,et al.  Planen: Einführung in die Planerstellungsmethoden der künstlichen Intelligenz , 1989, Reihe Informatik.

[31]  Eric Werner,et al.  What Can Agents Do Together? A Semantics for Reasoning about Cooperative Ability , 1990, ECAI.

[32]  Matthias Hecking.  HOW TO USE PLAN RECOGNITION TO IMPROVE THE ABILITIES OF THE INTELLIGENT HELP SYSTEM SINIX CONSULTANT , 1987 .

[33]  Evangelos Kranakis.  Towards a General Theory , 1986 .

[34]  Jerome Azarewicz,et al.  Plan Recognition for Airborne Tactical Decision Making , 1986, AAAI.

[35]  Hans-Hellmut Nagel,et al.  From image sequences towards conceptual descriptions , 1988, Image Vis. Comput..

[36]  Henry Kautz,et al.  A circumscriptive theory of plan recognition , 1990 .

[37]  Robert Wilensky,et al.  Understanding Goal-Based Stories , 1978, Outstanding Dissertations in the Computer Sciences.

[38]  A. Goldman.  Theory of Human Action , 1970 .

[39]  G. Miller,et al.  Plans and the structure of behavior , 1960 .

[40]  William J. Clancey,et al.  Plan Recognition Strategies in Student Modeling: Prediction and Description , 1982, AAAI.

[41]  Philip R. Cohen,et al.  Plans as Complex Mental Attitudes , 2003 .

[42]  James F. Allen.  Towards a General Theory of Action and Time , 1984, Artif. Intell..

[43]  Sandra Carberry,et al.  Incorporating Default Inferences Into Plan Recognition , 1990, AAAI.

[44]  C. Sidner,et al.  Plans for Discourse , 1988 .

[45]  Gerhard Fischer,et al.  Knowledge-based help systems , 1985, CHI '85.

[46]  Eric Werner,et al.  Cooperating Agents: A Unified Theory of Communication and Social Structure , 1989, Distributed Artificial Intelligence.

[47]  Timothy W. Finin.  Providing Help and Advice in Task Oriented Systems , 1983, IJCAI.

[48]  G. Miller,et al.  Plans and the structure of behavior , 1960 .

[49]  Bertram C. Bruce.  Plans and Social Actions , 1977 .

[50]  R. Wilensky.  Planning and Understanding: A Computational Approach to Human Reasoning , 1983 .

[51]  Gudula Retz-Schmidt.  Die Interpretation des Verhaltens mehrerer Akteure in Szenenfolgen , 1992, Informatik-Fachberichte.

[52]  Jeffrey S. Rosenschein,et al.  Rational interaction: cooperation among intelligent agents , 1986 .

[53]  Henry A. Kautz.  A formal theory of plan recognition , 1987 .

[54]  Victor R. Lesser,et al.  Focusing in Plan Recognition , 1984, AAAI.

[55]  A. Baier,et al.  Act and Intent , 1970 .

[56]  Gudula Retz-Schmidt,et al.  A REPLAI of SOCCER: Recognizing Intentions in the Domain of Soccer Games , 1988, ECAI.

[57]  A. Kenny.  Action, Emotion And Will , 1965 .

[58]  Michael E. Bratman,et al.  Intention, Plans, and Practical Reason , 1991 .

[59]  Wolfgang Wahlster,et al.  Incremental Natural Language Description of Dynamic Imagery , 1989, Wissensbasierte Systeme.

[60]  P. Cohen.  Intention = Choice + Commitment1 , 2022 .

[61]  William F. Brewer,et al.  Use of plan schemata in the recall and recognition of goal-directed actions. , 1983 .

[62]  Harold R. Dickman,et al.  The Perception of Behavioral Units. , 1963 .

[63]  Thomas Rist,et al.  On the Simultaneous Interpretation of Real World Image Sequences and their Natural Language Description: The System Soccer , 1988, ECAI.

[64]  Michael R. Genesereth.  The Role of Plans in Automated Consultation , 1979, IJCAI.

[65]  William F. Brewer,et al.  Memory for goal-directed events , 1980, Cognitive Psychology.

[66]  Martha E. Pollack,et al.  A Model of Plan Inference That Distinguishes Between the Beliefs of Actors and Observers , 1986, ACL.

[67]  Elizabeth T. Whitaker,et al.  Plan recognition in intelligent tutoring systems , 1990, Intell. Tutoring Media.

[68]  J.F. Allen,et al.  Plans, goals, and language , 1986, Proceedings of the IEEE.