Modeling of spoken dialogue with and without visual information

Auditory information is the major factor in human communication but in practical conversations, visual information such as gesture, facial expression, and head movement clearly makes it much smoother and more natural. Most research related to analysis of spoken dialogue is based on only auditory information. We are trying to clarify how humans utilize the knowledge of spoken dialogue management by dealing with more natural communication that includes visual information. Above all, by using visual information we can deal with a listener's attitude to the speaker-this cannot be done by using only auditory information.