Investigation of Methods to Create Future Multimodal Emotional Data for Robot Interactions in Patients with Schizophrenia: A Case Study

Rapid progress in humanoid robot investigations offers possibilities for improving the competencies of people with social disorders, although this improvement of humanoid robots remains unexplored for schizophrenic people. Methods for creating future multimodal emotional data for robot interactions were studied in this case study of a 40-year-old male patient with disorganized schizophrenia without comorbidities. The qualitative data included heart rate variability (HRV), video-audio recordings, and field notes. HRV, Haar cascade classifier (HCC), and Empath API© were evaluated during conversations between the patient and robot. Two expert nurses and one psychiatrist evaluated facial expressions. The research hypothesis questioned whether HRV, HCC, and Empath API© are useful for creating future multimodal emotional data about robot–patient interactions. The HRV analysis showed persistent sympathetic dominance, matching the human–robot conversational situation. The result of HCC was in agreement with that of human observation, in the case of rough consensus. In the case of observed results disagreed upon by experts, the HCC result was also different. However, emotional assessments by experts using Empath API© were also found to be inconsistent. We believe that with further investigation, a clearer identification of methods for multimodal emotional data for robot interactions can be achieved for patients with schizophrenia.

[1]  Dr. T Vijay Muni,et al.  Deep Learning Techniques for Speech Emotion Recognition , 2022, 2022 International Conference on Futuristic Technologies (INCOFT).

[2]  A. Ultsch,et al.  Explainable Artificial Intelligence (XAI) in Biomedicine: Making AI Decisions Trustworthy for Physicians and Patients , 2021, BioMedInformatics.

[3]  Sarah M. Haigh,et al.  Heart Rate Variability in Schizophrenia and Autism , 2021, Frontiers in Psychiatry.

[4]  T. Tanioka,et al.  Intentional Observational Clinical Research Design: Innovative Design for Complex Clinical Research Using Advanced Technology , 2021, International journal of environmental research and public health.

[5]  Qiuhong Li,et al.  Auditory emotion recognition deficits in schizophrenia: A systematic review and meta-analysis. , 2021, Asian journal of psychiatry.

[6]  Noah R. Flynn,et al.  Machine learning liver-injuring drug interactions with non-steroidal anti-inflammatory drugs (NSAIDs) from a retrospective electronic health record (EHR) cohort , 2021, PLoS Comput. Biol..

[7]  Yong Xu,et al.  Facial Emotion Recognition in Schizophrenia , 2021, Frontiers in Psychiatry.

[8]  G. Remington,et al.  Autonomic nervous system dysfunction in schizophrenia: impact on cognitive and metabolic health , 2021, npj Schizophrenia.

[9]  T. Tanioka,et al.  Development Issues of Healthcare Robots: Compassionate Communication for Older Adults with Dementia , 2021, International journal of environmental research and public health.

[10]  M. Breakspear,et al.  An active inference perspective on the negative symptoms of schizophrenia. , 2021, The lancet. Psychiatry.

[11]  P A Riyantoko,et al.  Facial Emotion Detection Using Haar-Cascade Classifier and Convolutional Neural Networks , 2021 .

[12]  Ludovic Marin,et al.  Study of Coordination Between Patients with Schizophrenia and Socially Assistive Robot During Physical Activity , 2021, Int. J. Soc. Robotics.

[13]  Fan Zhang,et al.  Speech Emotion Recognition with Multiscale Area Attention and Data Augmentation , 2021, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Daniel Sierra-Sosa,et al.  Deep Learning Techniques for Speech Emotion Recognition, from Databases to Models , 2021, Sensors.

[15]  P. Jackson,et al.  Emotion regulation of others’ positive and negative emotions is related to distinct patterns of heart rate variability and situational empathy , 2020, PloS one.

[16]  S. Joshua Swamidass,et al.  ‘Black Box’ to ‘Conversational’ Machine Learning: Ondansetron Reduces Risk of Hospital-Acquired Venous Thromboembolism , 2020, IEEE Journal of Biomedical and Health Informatics.

[17]  J. Lindenmayer,et al.  The psychometric properties of the Self-Evaluation of Negative Symptoms Scale (SNS) in treatment-resistant schizophrenia (TRS) , 2020, Schizophrenia Research.

[18]  O. Andreassen,et al.  Heart rate variability is associated with disease severity in psychosis spectrum disorders , 2020, Progress in Neuro-Psychopharmacology and Biological Psychiatry.

[19]  Yue-jia Luo,et al.  How Do Amusement, Anger and Fear Influence Heart Rate and Heart Rate Variability? , 2019, Front. Neurosci..

[20]  Evangelos D. Spyrou,et al.  Emotion Recognition from Speech: A Survey , 2019, WEBIST.

[21]  B. Hatef,et al.  The persistent effect of acute psychosocial stress on heart rate variability , 2019, The Egyptian Heart Journal.

[22]  S. Schoenhofer,et al.  Engaging Robots as Nursing Partners in Caring: Nursing as Caring Meets Care-Centered Value-Sensitive Design , 2019, International Journal for Human Caring.

[23]  T. Lincoln,et al.  Specificity of resting-state heart rate variability in psychosis: A comparison with clinical high risk, anxiety, and healthy controls , 2019, Schizophrenia Research.

[24]  Sakiko Yoshikawa,et al.  Facial Expressions of Basic Emotions in Japanese Laypeople , 2019, Front. Psychol..

[25]  Peter J. Gianaros,et al.  Should heart rate variability be “corrected” for heart rate? Biological, quantitative, and interpretive considerations , 2018, Psychophysiology.

[26]  B. Colombo,et al.  Cardiac vagal control as a marker of emotion regulation in healthy adults: A review , 2017, Biological Psychology.

[27]  A. Billard,et al.  Influence of facial feedback during a cooperative human-robot task in schizophrenia , 2017, Scientific Reports.

[28]  F. Shaffer,et al.  An Overview of Heart Rate Variability Metrics and Norms , 2017, Front. Public Health.

[29]  Aude Billard,et al.  Humanoid robots versus humans: How is emotional valence of facial expressions recognized by individuals with schizophrenia? An exploratory study , 2016, Schizophrenia Research.

[30]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[31]  Michael S. Bernstein,et al.  Empath: Understanding Topic Signals in Large-Scale Text , 2016, CHI.

[32]  J. Thayer,et al.  Resting vagal activity in schizophrenia: Meta-analysis of heart rate variability as a potential endophenotype , 2016, British Journal of Psychiatry.

[33]  Mar Rus-Calafell,et al.  A virtual reality-integrated program for improving social skills in patients with schizophrenia: a pilot study. , 2014, Journal of behavior therapy and experimental psychiatry.

[34]  Kazunori Matsumoto,et al.  Emotional processing during speech communication and positive symptoms in schizophrenia , 2013, Psychiatry and clinical neurosciences.

[35]  Franz Resch,et al.  Facial emotional expression in schizophrenia adolescents during verbal interaction with a parent , 2013, European Archives of Psychiatry and Clinical Neuroscience.

[36]  M. Kubicki,et al.  Schizophrenia as a Disorder of Communication , 2013, Schizophrenia research and treatment.

[37]  H. Fatouros-Bergman,et al.  Stability of Facial Affective Expressions in Schizophrenia , 2012, Schizophrenia research and treatment.

[38]  Steven Skiena,et al.  Empath: A framework for evaluating entity-level sentiment analysis , 2011, 2011 8th International Conference & Expo on Emerging Technologies for a Smarter World.

[39]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[40]  Mircea Miclea,et al.  Reduced heart rate variability and vagal tone in anxiety: Trait versus state, and the effects of autogenic training , 2009, Autonomic Neuroscience.

[41]  Philip D. Harvey,et al.  Communication abnormalities predict functional outcomes in chronic schizophrenia: Differential associations with social and adaptive functions , 2008, Schizophrenia Research.

[42]  D. Penn,et al.  Emotion perception and social skill over the course of psychosis: A comparison of individuals “at-risk” for psychosis and individuals with early and chronic schizophrenia spectrum illness , 2007, Cognitive neuropsychiatry.

[43]  David L. Roberts,et al.  The functional significance of social cognition in schizophrenia: a review. , 2006, Schizophrenia bulletin.

[44]  Akshay Deepak,et al.  A survey of speech emotion recognition in natural environment , 2021, Digit. Signal Process..

[45]  A. Leszczynska.  Facial emotion perception and schizophrenia symptoms. , 2015, Psychiatria polska.

[46]  Wei Xiao-peng,et al.  Survey on speech emotion recognition , 2009 .