How to Build a Supervised Autonomous System for Robot-Enhanced Therapy for Children with Autism Spectrum Disorder

Abstract Robot-Assisted Therapy (RAT) has successfully been used to improve social skills in children with autism spectrum disorders (ASD) through remote control of the robot in so-called Wizard of Oz (WoZ) paradigms.However, there is a need to increase the autonomy of the robot both to lighten the burden on human therapists (who have to remain in control and, importantly, supervise the robot) and to provide a consistent therapeutic experience. This paper seeks to provide insight into increasing the autonomy level of social robots in therapy to move beyond WoZ. With the final aim of improved human-human social interaction for the children, this multidisciplinary research seeks to facilitate the use of social robots as tools in clinical situations by addressing the challenge of increasing robot autonomy.We introduce the clinical framework in which the developments are tested, alongside initial data obtained from patients in a first phase of the project using a WoZ set-up mimicking the targeted supervised-autonomy behaviour. We further describe the implemented system architecture capable of providing the robot with supervised autonomy.

[1]  Fei-FeiLi,et al.  Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words , 2008 .

[2]  Tony Belpaeme,et al.  SPARC: an efficient way to combine reinforcement learning and supervised autonomy , 2016 .

[3]  Honghai Liu,et al.  Combining 3D joints Moving Trend and Geometry property for human action recognition , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[4]  Wanqing Li,et al.  Action recognition based on a bag of 3D points , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[5]  P. Ekman,et al.  Facial Action Coding System: Manual , 1978 .

[6]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[7]  Ambarish Goswami,et al.  Hardware experiments of humanoid robot safe fall using Aldebaran NAO , 2012, 2012 IEEE International Conference on Robotics and Automation.

[8]  Serge Thill,et al.  Engagement: A traceable motivational concept in human-robot interaction , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[9]  Erhardt Barth,et al.  Accurate Eye Centre Localisation by Means of Gradients , 2011, VISAPP.

[10]  C. Corsello,et al.  Early Intervention in Autism , 2005 .

[11]  Ting Wang,et al.  Combining Kinect and PnP for camera pose estimation , 2015, 2015 8th International Conference on Human System Interaction (HSI).

[12]  Youfu Li,et al.  Entropy distribution and coverage rate-based birth intensity estimation in GM-PHD filter for multi-target visual tracking , 2014, Signal Process..

[13]  Ivan Tashev.  Recent Advances in Human-Machine Interfaces for Gaming and Entertainment , 2011 .

[14]  Tsukasa Ogasawara,et al.  Humanoid with Interaction Ability Using Vision and Speech Information , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Bhiksha Raj,et al.  Microphone array processing for distant speech recognition: Towards real-world deployment , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[16]  Atsuo Takanishi,et al.  Whole body emotion expressions for KOBIAN humanoid robot — preliminary experiments with different Emotional patterns — , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[17]  Bram Vanderborght,et al.  A multilayer reactive system for robots interacting with children with autism , 2016, ArXiv.

[18]  Tony Belpaeme,et al.  SPARC: Supervised Progressively Autonomous Robot Competencies , 2015, ICSR.

[19]  Haizhou Li,et al.  An overview of text-independent speaker recognition: From features to supervectors , 2010, Speech Commun..

[20]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Emilia I. Barakova,et al.  Expressing and interpreting emotional movements in social games with robots , 2010, Personal and Ubiquitous Computing.

[22]  D K Oller,et al.  Automated vocal analysis of naturalistic recordings from children with autism, language delay, and typical development , 2010, Proceedings of the National Academy of Sciences.

[23]  D. Lefeber,et al.  Using the social robot probo as a social story telling agent for children with ASD , 2012 .

[24]  Jerry Alan Fails,et al.  Interactive machine learning , 2003, IUI '03.

[25]  A. Young,et al.  Emotion Perception from Dynamic and Static Body Expressions in Point-Light and Full-Light Displays , 2004, Perception.

[26]  B. Leventhal,et al.  The Autism Diagnostic Observation Schedule—Generic: A Standard Measure of Social and Communication Deficits Associated with the Spectrum of Autism , 2000, Journal of autism and developmental disorders.

[27]  Kazuhito Yokoi,et al.  UKEMI: falling motion control to minimize damage to biped humanoid robot , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  M. Rutter,et al.  Adult outcome for children with autism. , 2004, Journal of child psychology and psychiatry, and allied disciplines.

[29]  I. William Zartman.  The Southwest Shore , 1972 .

[30]  François Michaud,et al.  Characteristics of mobile robotic toys for children with pervasive developmental disorders , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[31]  Honghai Liu,et al.  Tracking Multiple Video Targets with an Improved GM-PHD Tracker , 2015, Sensors.

[32]  Jorge Dias,et al.  Attentional Mechanisms for Socially Interactive Robots–A Survey , 2014, IEEE Transactions on Autonomous Mental Development.

[33]  Marcelo H. Ang,et al.  Why Robots? A Survey on the Roles and Benefits of Social Robots in the Therapy of Children with Autism , 2013, International Journal of Social Robotics.

[34]  Hubert Korzilius,et al.  A meta-analytic study on the effectiveness of comprehensive ABA-based early intervention programs for children with Autism Spectrum Disorders , 2011 .

[35]  Fabio Tesser,et al.  Multimodal child-robot interaction: building social bonds , 2013, HRI 2013.

[36]  Richard P. Hastings,et al.  Meta-Analysis of Early Intensive Behavioral Intervention for Children With Autism , 2009, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[37]  B. Scassellati,et al.  Robots for use in autism research. , 2012, Annual review of biomedical engineering.

[38]  Guido Bugmann,et al.  Modeling the Human Blink: a Computational Model for Use within Human-Robot Interaction , 2013, Int. J. Humanoid Robotics.

[39]  F. Volkmar,et al.  Speech and prosody characteristics of adolescents and adults with high-functioning autism and Asperger syndrome. , 2001, Journal of speech, language, and hearing research : JSLHR.

[40]  Danilo De Rossi,et al.  Designing and Evaluating a Social Gaze-Control System for a Humanoid Robot , 2014, IEEE Transactions on Human-Machine Systems.

[41]  Jorge Dias,et al.  Designing an artificial attention system for social robots , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[43]  Ronald C. Arkin.  Homeostatic control for a mobile robot: Dynamic replanning in hazardous environments , 1992, J. Field Robotics.

[44]  Juan Carlos Niebles,et al.  A Hierarchical Model of Shape and Appearance for Human Action Classification , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Bram Vanderborght,et al.  A Survey of Expectations About the Role of Robots in Robot-Assisted Therapy for Children with ASD: Ethical Acceptability, Trust, Sociability, Appearance, and Attachment , 2015, Science and Engineering Ethics.

[46]  Bram Vanderborght,et al.  Development of a generic method to generate upper-body emotional expressions for different social robots , 2015, Adv. Robotics.

[47]  Kerstin Dautenhahn,et al.  Therapeutic and educational objectives in robot assisted play for children with autism , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[48]  Tom Ziemke,et al.  Robot-assisted therapy for autism spectrum disorders with (partially) autonomous control: Challenges and outlook , 2012, Paladyn J. Behav. Robotics.

[49]  Honghai Liu,et al.  Visual Focus of Attention Estimation Using Eye Center Localization , 2017, IEEE Systems Journal.

[50]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Adriana Tapus,et al.  Individuals with Autism: Analysis of the First Interaction with Nao Robot Based on Their Proprioceptive and Kinematic Profiles , 2015, RAAD.

[52]  Adriana Tapus,et al.  User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy , 2008, Intell. Serv. Robotics.

[53]  Arnstein Mykletun,et al.  Is Long-Term Prognosis for Pervasive Developmental Disorder Not Otherwise Specified Different from Prognosis for Autistic Disorder? Findings from a 30-Year Follow-Up Study , 2012, Journal of autism and developmental disorders.

[54]  Bram Vanderborght,et al.  Reaching and pointing gestures calculated by a generic gesture system for social robots , 2016, Robotics Auton. Syst..

[55]  B. Robins,et al.  Does appearance matter in the interaction of children with autism with a humanoid robot , 2006 .

[56]  T. Takenaka,et al.  The development of Honda humanoid robot , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[57]  Tony Belpaeme,et al.  Human-Guided Learning of Social Action Selection for Robot-Assisted Therapy , 2015, MLIS@ICML.

[58]  Kerstin Dautenhahn,et al.  Eliciting Requirements for a Robotic Toy for Children with Autism - Results from User Panels , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[59]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[60]  Donald A. Norman,et al.  Affect and machine design: Lessons for the development of autonomous machines , 2003, IBM Syst. J..

[61]  Tony Belpaeme,et al.  A touchscreen-based ‘Sandtray’ to facilitate, mediate and contextualise human-robot social interaction , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[62]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[63]  Andrea Lockerd Thomaz,et al.  Teachable robots: Understanding human teaching behavior to build more effective robot learners , 2008, Artif. Intell..

[64]  JongSuk Choi,et al.  A robot-assisted behavioral intervention system for children with autism spectrum disorders , 2016, Robotics Auton. Syst..

[65]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[66]  A. Sloman.  Beyond Shallow Models of Emotion , 2001 .

[67]  Takayuki Kanda,et al.  Natural deictic communication with humanoid robots , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[68]  Kamlesh Lakhwani,et al.  Optimization Of Frame Rate In Real Time Object Detection And Tracking , 2013 .

[69]  Abraham S. Luchins,et al.  The Relation of Size of Light to Autokinetic Effect , 1954 .

[70]  S. Ozonoff.  Reliability and validity of the Wisconsin Card Sorting Test in studies of autism. , 1995 .

[71]  B. Ingersoll,et al.  The Social Role of Imitation in Autism: Implications for the Treatment of Imitation Deficits , 2008 .

[72]  Lauren M. Schmitt,et al.  The Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review. , 2012, Research in autism spectrum disorders.

[73]  Mark Coeckelbergh,et al.  Are Emotional Robots Deceptive? , 2012, IEEE Transactions on Affective Computing.

[74]  Bram Vanderborght,et al.  “Can You Cure me? Children With Autism Spectrum Disorders Playing a Doctor Game With a Social Robot” , 2016 .

[75]  Juan Carlos Niebles,et al.  Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words , 2006, BMVC.

[76]  Larry S. Davis,et al.  Model-based object pose in 25 lines of code , 1992, International Journal of Computer Vision.

[77]  Jordi Albo-Canals,et al.  Modelling social skills and problem solving strategies used by children with ASD through cloud connected social robots as data loggers: first modelling approach , 2016 .

[78]  Honghai Liu,et al.  Gaze estimation driven solution for interacting children with ASD , 2015, 2015 International Symposium on Micro-NanoMechatronics and Human Science (MHS).

[79]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[80]  Thomas K. Landauer,et al.  Psychology as a mother of invention , 1986, CHI '87.

[81]  Youfu Li,et al.  GM-PHD-Based Multi-Target Visual Tracking Using Entropy Distribution and Game Theory , 2014, IEEE Transactions on Industrial Informatics.

[82]  A. Takanishi,et al.  Various emotional expressions with emotion expression humanoid robot WE-4RII , 2004, IEEE Conference on Robotics and Automation, 2004. TExCRA Technical Exhibition Based..

[83]  Jason M. O'Kane,et al.  A low-cost socially assistive robot and robot-assisted intervention for children with autism spectrum disorder: field trials and lessons learned , 2016, Autonomous Robots.

[84]  Aaron Sloman,et al.  Varieties of Metacognition in Natural and Artificial Systems , 2011, Metareasoning.

[85]  Claire A. G. J. Huijnen,et al.  Mapping Robots to Therapy and Educational Objectives for Children with Autism Spectrum Disorder , 2016, Journal of Autism and Developmental Disorders.

[86]  Honghai Liu,et al.  Convolution-based means of gradient for fast eye center localization , 2015, 2015 International Conference on Machine Learning and Cybernetics (ICMLC).

[87]  Christina de Rivera,et al.  The Use of Intensive Behavioural Intervention for Children With Autism , 2008 .

[88]  Larry S. Davis,et al.  Model-Based Object Pose in 25 Lines of Code , 1992, ECCV.

[89]  Jing Li,et al.  An Integrative Framework of Human Hand Gesture Segmentation for Human–Robot Interaction , 2017, IEEE Systems Journal.

[90]  Andreea Peca,et al.  Robot Enhanced Therapy for Children with Autism Disorders: Measuring Ethical Acceptability , 2016, IEEE Technology and Society Magazine.

[91]  Matti Pietikäinen,et al.  Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[92]  Brett Stevens,et al.  Dynamic facial expression recognition using local patch and LBP-TOP , 2015, 2015 8th International Conference on Human System Interaction (HSI).

[93]  Maya Cakmak,et al.  Power to the People: The Role of Humans in Interactive Machine Learning , 2014, AI Mag..

[94]  Bram Vanderborght,et al.  Expressing Emotions with the Social Robot Probo , 2010, Int. J. Soc. Robotics.

[95]  Abdul Rahman Omar,et al.  ASKNAO apps targeting at social skills development for children with autism , 2015, 2015 IEEE International Conference on Automation Science and Engineering (CASE).

[96]  Daniel David,et al.  Robot-Based Psychotherapy: Concepts Development, State of the Art, and New Directions , 2014 .

[97]  Susanne A. Denham,et al.  Response: Spinning the Pinwheel, Together: More Thoughts on Affective Social Competence. , 2001 .

[98]  Maja J. Matarić,et al.  B3IA: A control architecture for autonomous robot-assisted behavior intervention for children with Autism Spectrum Disorders , 2008, RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication.

[99]  J. Pina-Cabral,et al.  Tamed Violence: Genital Symbolism in Portuguese Popular Culture , 1993 .

[100]  James Wilson,et al.  Rapid Prototyping for User Interface Design , 1988 .