Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face

A new integrated method is presented to recognize the emotional expressions of human using both voices and facial expressions. For voices, we use such prosodic parameters as pitch signals, energy, and their derivatives, which are trained by hidden Markov model for recognition. For facial expressions, we use feature parameters from thermal images in addition to visible images, which are trained by neural networks for recognition. The thermal images are observed by infrared ray which is not influenced by lighting conditions. The total recognition rates show better performance than that obtained from each single experiment. The results are compared with the recognition by human questionnaire.