Voice and gesture recognition system facilitating communication between man and virtual agent

In this research, a dialog environment between human and virtual environment has been constructed. At preset, if a person wants to interact with virtual environment, special device such a data glove is required, but it makes difficult for general users to manipulate virtual objects. When we cannot manipulate objects directly, it is natural that we ask someone with a privilege to do the operation in place of us. In the case, it is convenient if methods used in daily life are allowed. That is, the method to be proposed here is based on the integration between verbal information through the utterances and non-verbal information by the gestures such as finger pointing. The experimental results have proved the effectiveness of this approach in terms of facilitating man-machine interaction and communication. The environment constructed in this research allows a user to communicate by talking and showing gestures to a personified agent in virtual environment. A user can use his/her finger to point at a virtual object and ask him to manipulate it.