Toward Personalized Affect-Aware Socially Assistive Robot Tutors in Long-Term Interventions for Children with Autism

Affect-aware socially assistive robotics (SAR) has shown great potential for augmenting interventions for children with autism spectrum disorders (ASD). However, current SAR cannot yet perceive the unique and diverse set of atypical cognitive-affective behaviors from children with ASD in an automatic and personalized fashion in long-term (multi-session) real-world interactions. To bridge this gap, this work designed and validated personalized models of arousal and valence for children with ASD using a multi-session in-home dataset of SAR interventions. By trainingmachine learning (ML) algorithmswith supervised domain adaptation (s-DA), the personalized models were able to trade off between the limited individual data and the more abundant less personal data pooled from other study participants. We evaluated the effects of personalization on a long-term multimodal dataset consisting of 4 children with ASD with a total of 19 sessions, and derived inter-rater reliability (IR) scores for binary arousal (IR = 83%) and valence (IR = 81%) labels between human annotators. Our results show that personalized Gradient Boosted Decision Trees (XGBoost) models with s-DA outperformed two non-personalized individualized and generic model baselines not only on the weighted average of all sessions, but also statistically (p < .05) across individual sessions. This work paves the way for the development of personalized autonomous SAR systems tailored toward individuals with atypical cognitive-affective and socio-emotional needs.

[1]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[2]  C. Kasari,et al.  SMARTer Approach to Personalizing Intervention for Children With Autism Spectrum Disorder. , 2018, Journal of speech, language, and hearing research : JSLHR.

[3]  Peter Bailis,et al.  CrossTrainer: Practical Domain Adaptation with Loss Reweighting , 2019, DEEM@SIGMOD.

[4]  Hatice Gunes,et al.  Automatic, Dimensional and Continuous Emotion Recognition , 2010, Int. J. Synth. Emot..

[5]  Maureen S. Durkin,et al.  Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years — Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2012 , 2018, Morbidity and mortality weekly report. Surveillance summaries.

[6]  Conor McGinn,et al.  Using the Geneva Emotion Wheel to Measure Perceived Affect in Human-Robot Interaction , 2020, 2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[7]  Arthur C. Graesser,et al.  Toward an Affect-Sensitive AutoTutor , 2007, IEEE Intelligent Systems.

[8]  Cynthia Breazeal,et al.  Frustratingly Easy Personalization for Real-time Affect Interpretation of Facial Expression , 2019, 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII).

[9]  Marcelo H. Ang,et al.  Why Robots? A Survey on the Roles and Benefits of Social Robots in the Therapy of Children with Autism , 2013, International Journal of Social Robotics.

[10]  H. Gunes,et al.  Live human–robot interactive public demonstrations with automatic emotion and personality prediction , 2019, Philosophical Transactions of the Royal Society B.

[11]  Beverly Park Woolf,et al.  Affect-aware tutors: recognising and responding to student affect , 2009, Int. J. Learn. Technol..

[12]  B. Scassellati,et al.  Robots for use in autism research. , 2012, Annual review of biomedical engineering.

[13]  Brian Scassellati,et al.  Improving social skills in children with ASD using a long-term, in-home social robot , 2018, Science Robotics.

[14]  G. Arbanas.  Diagnostic and Statistical Manual of Mental Disorders (DSM-5) , 2015 .

[15]  Sonja A. Kotz,et al.  Valence, arousal, and task effects in emotional prosody processing , 2013, Front. Psychol..

[16]  Cynthia Breazeal,et al.  Affect-Aware Student Models for Robot Tutors , 2016, AAMAS.

[17]  J. Burack,et al.  Sensory Characteristics in ASD , 2009, McGill journal of medicine : MJM : an international forum for the advancement of medical sciences by students.

[18]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[19]  Chung Hyuk Park,et al.  Toward an Automated Measure of Social Engagement for Children With Autism Spectrum Disorder—A Personalized Computational Modeling Approach , 2020, Frontiers in Robotics and AI.

[20]  Maja J. Matarić,et al.  Attentiveness of Children with Diverse Needs to a Socially Assistive Robot in the Home , 2018 .

[21]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[22]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[23]  Gisele Ragusa,et al.  Long-Term Personalization of an In-Home Socially Assistive Robot for Children With Autism Spectrum Disorders , 2019, Front. Robot. AI.

[24]  Sidney D'Mello,et al.  The half-life of cognitive-affective states during complex learning , 2011, Cognition & emotion.

[25]  H. Gunes,et al.  Computational Analysis of Affect, Personality, and Engagement in Human–Robot Interactions ⁎ ⁎The research reported in this chapter was completed while O. Celiktutan and E. Sariyanidi were with the Computer Laboratory, University of Cambridge, United Kingdom. , 2018 .

[26]  D. Feil-Seifer,et al.  Defining socially assistive robotics , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[27]  Brian Scassellati,et al.  The Effect of Personalization in Longer-Term Robot Tutoring , 2018, ACM Trans. Hum. Robot Interact..

[28]  Rosalind W. Picard,et al.  An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[29]  Björn W. Schuller,et al.  CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[30]  G. Geffken,et al.  A Survey of Autism Knowledge in a Health Care Setting , 2005, Journal of autism and developmental disorders.

[31]  Björn W. Schuller,et al.  Measuring Engagement in Robot-Assisted Autism Therapy: A Cross-Cultural Study , 2017, Front. Robot. AI.

[32]  Joni Dambre,et al.  Leveraging Robotics Research for Children with Autism: A Review , 2018, International Journal of Social Robotics.

[33]  Björn W. Schuller,et al.  Personalized machine learning for robot perception of affect and engagement in autism therapy , 2018, Science Robotics.

[34]  Bilge Mutlu,et al.  Learning-Based Modeling of Multimodal Behaviors for Humanlike Robots , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[35]  Ana Paiva,et al.  Detecting user engagement with a robot companion using task and social interaction-based features , 2009, ICMI-MLMI '09.

[36]  Balasubramanian Thiagarajan,et al.  Modeling engagement in long-term, in-home socially assistive robot interventions for children with autism spectrum disorders , 2020, Science Robotics.

[37]  Caitlyn Clabaugh,et al.  Escaping Oz: Autonomy in Socially Assistive Robotics , 2019, Annu. Rev. Control. Robotics Auton. Syst..

[38]  Shaobo Huang,et al.  How to train your DragonBot: Socially assistive robots for teaching children about nutrition through play , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[39]  Maja J. Mataric,et al.  Socially Assistive Robotics for Personalized Education for Children , 2014, AAAI Fall Symposia.

[40]  M. R. Lepper,et al.  Socializing the intelligent tutor: bringing empathy to computer tutors , 1988 .

[41]  Tatsuya Kawahara,et al.  Detection of social signals for recognizing engagement in human-robot interaction , 2017, ArXiv.

[42]  Nicu Sebe,et al.  Unsupervised Domain Adaptation for Personalized Facial Emotion Recognition , 2014, ICMI.

[43]  Cynthia Breazeal,et al.  Personalized Estimation of Engagement From Videos Using Active Learning With Deep Reinforcement Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[44]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[45]  Elsevier Sdol.  International Journal of Human-Computer Studies , 2009 .

[46]  Maja J. Mataric,et al.  Designing a Socially Assistive Robot for Long-Term In-Home Use for Children with Autism Spectrum Disorders , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[47]  Diane J. Cook,et al.  A Survey of Unsupervised Deep Domain Adaptation , 2018, ACM Trans. Intell. Syst. Technol..

[48]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[49]  Ana Paiva,et al.  Automatic analysis of affective postures and body motion to detect engagement with a game companion , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[50]  D Feil-Seifer,et al.  Socially Assistive Robotics , 2011, IEEE Robotics & Automation Magazine.

[51]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[52]  Kate Saenko,et al.  Return of Frustratingly Easy Domain Adaptation , 2015, AAAI.

[53]  Louis-Philippe Morency,et al.  OpenFace 2.0: Facial Behavior Analysis Toolkit , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[54]  Cynthia Breazeal,et al.  Emotion and sociable humanoid robots , 2003, Int. J. Hum. Comput. Stud..

[55]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .