Social Interaction: Multimodal Conversation with Social Agents

We present a new approach to human-computer interaction, called social interaction. Its main characteristics are summarized by the following three points. First, interactions are realized as multimodal (verbal and nonverbal) conversation using spoken language, facial expressions, and so on. Second, the conversants are a group of humans and social agents that are autonomous and social. Autonomy is an important property that allows agents to decide how to act in an ever-changing environment. Socialness is also an important property that allows agents to behave both cooperatively and collaboratively. Generally, conversation is a joint work and ill-structured. Its participants are required to be social as well as autonomous. Third, conversants often encounter communication mismatches (misunderstanding others' intentions and beliefs) and fail to achieve their joint goals. The social agents, therefore, are always concerned with detecting communication mismatches. We realize a social agent that hears human-to-human conversation and informs what is causing the misunderstanding. It can also interact with humans by voice with facial displays and head (and eye) movement.

[1]  Akikazu Takeuchi,et al.  Communicative facial displays as a new conversational modality , 1993, INTERCHI.

[2]  Akikazu Takeuchi.  A Rapid Face Construction Lab , 1992 .

[3]  Keith Waters,et al.  A muscle model for animation three-dimensional facial expression , 1987, SIGGRAPH.

[4]  Nicole Chovil.  Discourse‐oriented facial displays in conversation , 1991 .

[5]  I. Katunobu,et al.  Continuous speech recognition by context-dependent phonetic HMM and an efficient algorithm for finding N-Best sentence hypotheses , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[6]  John R. Searle,et al.  Speech Acts: An Essay in the Philosophy of Language , 1970 .

[7]  Kôiti Hasida,et al.  Understanding Spoken Natural Language with Omni-Directional Information Flow , 1993, IJCAI.

[8]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[9]  Peter van Beek,et al.  Resolving Plan Ambiguity for Cooperative Response Generation , 1991, IJCAI.

[10]  Katashi Nagao.  A Preferential Constraint Satisfaction Technique for Natural Language Analysis , 1992, ECAI.

[11]  Herbert A. Simon,et al.  Situated Action: A Symbolic Interpretation , 1993, Cogn. Sci..

[12]  L. Suchman.  Plans and situated actions , 1987 .

[13]  Katashi Nagao.  Abduction and Dynamic Preference in Plan-Based Dialogue Understanding , 1993, IJCAI.

[14]  Henry Kautz,et al.  A circumscriptive theory of plan recognition , 1990 .

[15]  Hozumi Tanaka,et al.  Continuous speech recognition by context-dependent phonetic HMM and an efficient algorithm for finding N-Best sentence hypotheses , 1992, ICASSP.

[16]  W. Scott Neal Reilly,et al.  An Architecture for Action, Emotion, and Social Behavior , 1992, MAAMAW.

[17]  K. Korta,et al.  Plan Recognition in Natural Language Dialogue , 1994 .