Patient State Recognition System for Healthcare Using Speech and Facial Expressions

Smart, interactive healthcare is necessary in the modern age. Several issues, such as accurate diagnosis, low-cost modeling, low-complexity design, seamless transmission, and sufficient storage, should be addressed while developing a complete healthcare framework. In this paper, we propose a patient state recognition system for the healthcare framework. We design the system in such a way that it provides good recognition accuracy, provides low-cost modeling, and is scalable. The system takes two main types of input, video and audio, which are captured in a multi-sensory environment. Speech and video input are processed separately during feature extraction and modeling; these two input modalities are merged at score level, where the scores are obtained from the models of different patients’ states. For the experiments, 100 people were recruited to mimic a patient’s states of normal, pain, and tensed. The experimental results show that the proposed system can achieve an average 98.2 % recognition accuracy.