Improving communication skills of children with autism through support of applied behavioral analysis treatments using multimedia computing: a survey

Naturalistic applied behavior analysis (ABA) techniques have been shown to help children with autism improve their communication skills. Recognizing that individuals who interact with children regularly are in the position to utilize treatments with profound effects, researchers have examined methodologies for training parents, teachers, and peers to implement treatments. These programs are time intensive and often unable to support trainees after training. Technologies need to be examined to determine how they can aid in the educational and support process. Academic publications and publicly available training programs were reviewed to determine the types of participants, methodologies, and training durations that have been reported for instructing interventionists. These resources illustrate a need to make programs more accessible. To address this, selected computer science research is applied to methods of evaluating ABA implementations in order to recommend how the technologies could be utilized to make training and support programs more accessible. Review results of instructional programs, both in research and available in the community, illustrate the challenges in providing training in ABA methodologies. Modern research in multimedia data processing and machine learning could be applied to reduce the human cost of training and support individuals implementing ABA techniques. Utilizing machine learning techniques to analyze video probes of naturalistic ABA treatment implementation could alleviate the human cost of evaluating fidelity, allowing for greater support for individuals interested in the treatments. These technologies could be used in the future to expand data collection to provide more perspective on the treatments.

[1]  K. Pierce,et al.  Increasing complex social behaviors in children with autism: effects of peer-implemented pivotal response training. , 1995, Journal of applied behavior analysis.

[2]  Helena Brentani,et al.  Procedures and compliance of a video modeling applied behavior analysis intervention for Brazilian parents of children with autism spectrum disorders , 2017, Autism : the international journal of research and practice.

[3]  S. Bryson,et al.  Brief parent training in pivotal response treatment for preschoolers with autism. , 2010, Journal of child psychology and psychiatry, and allied disciplines.

[4]  Nanning Zheng,et al.  Inferring Human Attention by Learning Latent Intentions , 2017, IJCAI.

[5]  L. Schreibman,et al.  Effects of sociodramatic play training on children with autism , 1995, Journal of autism and developmental disorders.

[6]  John H. L. Hansen,et al.  Unsupervised Speech Activity Detection Using Voicing Measures and Perceptual Spectral Flux , 2013, IEEE Signal Processing Letters.

[7]  Vittorio Murino,et al.  Social interactions by visual focus of attention in a three‐dimensional environment , 2013, Expert Syst. J. Knowl. Eng..

[8]  Tsuhan Chen,et al.  Spatio-Temporal Phrases for Activity Recognition , 2012, ECCV.

[9]  Rolf Baxter,et al.  An Adaptive Motion Model for Person Tracking with Instantaneous Head-Pose Features , 2015, IEEE Signal Processing Letters.

[10]  M. Frenn,et al.  Autism spectrum disorder: parenting stress, family functioning and health-related quality of life. , 2011, Families, systems & health : the journal of collaborative family healthcare.

[11]  Nazli Ikizler-Cinbis,et al.  Two-person interaction recognition via spatial multiple instance embedding , 2015, J. Vis. Commun. Image Represent..

[12]  Hisashi Osumi,et al.  Human Visual Attention Model Based on Analysis of Magic for Smooth Human–Robot Interaction , 2016, Int. J. Soc. Robotics.

[13]  Erik Marchi,et al.  Typicality and emotion in the voice of children with autism spectrum condition: evidence across three languages , 2015, INTERSPEECH.

[14]  Sethuraman Panchanathan,et al.  Parent and Child Voice Activity Detection in Pivotal Response Treatment Video Probes , 2019, HCI.

[15]  Manoj Kumar,et al.  Multi-Scale Context Adaptation for Improving Child Automatic Speech Recognition in Child-Adult Spoken Interactions , 2017, INTERSPEECH.

[16]  L. Vismara,et al.  Using Perseverative Interests to Elicit Joint Attention Behaviors in Young Children With Autism , 2007 .

[17]  Roberto Basili,et al.  Effective and Robust Natural Language Understanding for Human-Robot Interaction , 2014, ECAI.

[18]  J. Connell,et al.  A quantitative analysis of language interventions for children with autism. , 2010 .

[19]  Klaus Dorfmüller,et al.  Robust tracking for augmented reality using retroreflective markers , 1999, Comput. Graph..

[20]  Yifan Gong,et al.  An Overview of Noise-Robust Automatic Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[21]  Angel Fettig,et al.  Evidence-Based Practices for Children, Youth, and Young Adults with Autism Spectrum Disorder: A Comprehensive Review , 2015, Journal of autism and developmental disorders.

[22]  Yutong Zhang,et al.  An object tracking algorithm with embedded gyro information , 2017, International Conference on Electronics and Information Engineering.

[23]  Remco C. Veltkamp,et al.  Spatio-Temporal Detection of Fine-Grained Dyadic Human Interactions , 2016, HBU.

[24]  Gang Wang,et al.  NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  R. Koegel,et al.  The Use of a Self-Directed Learning Program to Provide Introductory Training in Pivotal Response Treatment to Parents of Children With Autism , 2010 .

[26]  A. Stahmer Teaching symbolic play skills to children with autism using Pivotal Response Training , 1995, Journal of autism and developmental disorders.

[27]  L. LeBlanc,et al.  Parent-implemented natural language paradigm to increase language and play in children with autism , 2007 .

[28]  J. C. C. Gillesen,et al.  From training to robot behavior: Towards custom scenarios for robotics in training programs for ASD , 2011, 2011 IEEE International Conference on Rehabilitation Robotics.

[29]  Gábor Gosztolya Detecting Laughter and Filler Events by Time Series Smoothing with Genetic Algorithms , 2016, SPECOM.

[30]  Fabien Ringeval,et al.  Speech-based Diagnosis of Autism Spectrum Condition by Generative Adversarial Network Representations , 2017, DH.

[31]  John H. L. Hansen,et al.  Signal processing for young child speech language development , 2008, WOCCI.

[32]  Ian Vince McLoughlin The use of low-frequency ultrasound for voice activity detection , 2014, INTERSPEECH.

[33]  Mohammad Rezaei,et al.  A Randomized Clinical Trial Comparison Between Pivotal Response Treatment (PRT) and Adult-Driven Applied Behavior Analysis (ABA) Intervention on Disruptive Behaviors in Public School Children with Autism , 2015, Journal of autism and developmental disorders.

[34]  M. Sherer,et al.  Individual behavioral profiles and predictors of treatment effectiveness for children with autism. , 2005, Journal of consulting and clinical psychology.

[35]  Joon-Hyuk Chang,et al.  Voice activity detection based on statistical models and machine learning approaches , 2010, Comput. Speech Lang..

[36]  Arthur Szlam,et al.  CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant , 2019, ArXiv.

[37]  L. Vismara,et al.  Preliminary Findings of a Telehealth Approach to Parent Training in Autism , 2013, Journal of autism and developmental disorders.

[38]  Antonio Y Hardan,et al.  A randomized controlled trial of Pivotal Response Treatment Group for parents of children with autism. , 2015, Journal of child psychology and psychiatry, and allied disciplines.

[39]  Shrikanth S. Narayanan,et al.  Acoustics of children's speech: developmental changes of temporal and spectral parameters. , 1999, The Journal of the Acoustical Society of America.

[40]  Yusuke Kida,et al.  Voice Activity Detection: Merging Source and Filter-based Information , 2016, IEEE Signal Processing Letters.

[41]  Yingguan Wang,et al.  Real-time scale-adaptive correlation filters tracker with depth information to handle occlusion , 2016, J. Electronic Imaging.

[42]  Qianli Xu,et al.  Attention-based addressee selection for service and social robots to interact with multiple persons , 2012, WASA '12.

[43]  L. Schreibman,et al.  Social Validation of Symbolic Play Training for Children with Autism. , 2006 .

[44]  Pamela Ventola,et al.  Improvements in Social and Adaptive Functioning Following Short-Duration PRT Program: A Clinical Replication , 2014, Journal of Autism and Developmental Disorders.

[45]  Helen Loeb,et al.  Stereo 3D tracking of infants in natural play conditions , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[46]  Jonathan Berant,et al.  Building a Semantic Parser Overnight , 2015, ACL.

[47]  L. Koegel,et al.  Teaching Children with Autism Self‐Initiations as a Pivotal Response , 2003 .

[48]  Stephen Camarata,et al.  Increasing Speech Intelligibility in Children with Autism , 1998, Journal of autism and developmental disorders.

[49]  Kristen Grauman,et al.  Efficient Activity Detection in Untrimmed Video with Max-Subgraph Search , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Aaron Albin,et al.  Automatic analysis of LENA recordings for language assessment in children aged five to fourteen years with application to individuals with autism , 2017, 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[51]  Agata Rozga,et al.  Joint Alignment and Modeling of Correlated Behavior Streams , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[52]  Antonio Y Hardan,et al.  Pivotal Response Treatment Parent Training for Autism: Findings from a 3-Month Follow-Up Evaluation , 2015, Journal of Autism and Developmental Disorders.

[53]  Hoifung Poon,et al.  Unsupervised Semantic Parsing , 2009, EMNLP.

[54]  David A. Forsyth,et al.  The Static Multimodal Dyadic Behavior Dataset for Engagement Prediction , 2016, ECCV Workshops.

[55]  R. Koegel,et al.  Parent Education for Families of Children with Autism Living in Geographically Distant Areas , 2002 .

[56]  Ana Paiva,et al.  Automatic analysis of affective postures and body motion to detect engagement with a game companion , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[57]  DeLiang Wang,et al.  Boosting Contextual Information for Deep Neural Network Based Voice Activity Detection , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[58]  Meng Zhang,et al.  Neural Network Methods for Natural Language Processing , 2017, Computational Linguistics.

[59]  Daniel Gildea,et al.  Automated prediction and analysis of job interview performance: The role of what you say and how you say it , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[60]  Greg Mori,et al.  Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Erik Cambria,et al.  Fusing audio, visual and textual clues for sentiment analysis from multimodal content , 2016, Neurocomputing.

[62]  S. Bryson,et al.  Effectiveness of Community-Based Early Intervention Based on Pivotal Response Treatment , 2015, Journal of Autism and Developmental Disorders.

[63]  L. Koegel,et al.  Improving Social Initiations in Young Children with Autism Using Reinforcers with Embedded Social Interactions , 2009, Journal of autism and developmental disorders.

[64]  Quoc V. Le,et al.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition , 2015, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[65]  Tara N. Sainath,et al.  Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[66]  L. Koegel,et al.  Pivotal Response Intervention I: Overview of Approach , 1999 .

[67]  L. Koegel,et al.  Brief Report: Question-Asking and Collateral Language Acquisition in Children with Autism , 2009, Journal of autism and developmental disorders.

[68]  L. Koegel,et al.  Pivotal Response Intervention II: Preliminary Long-Term Outcome Data , 1999 .

[69]  C. Harper,et al.  Recess is Time-in: Using Peers to Improve Social Skills of Children with Autism , 2008, Journal of autism and developmental disorders.

[70]  Michael C. Frank,et al.  Discovering the Signatures of Joint Attention in Child-Caregiver Interaction , 2014, CogSci.

[71]  Jessica Suhrheinrich,et al.  Exploring the Effect of Immediate Video Feedback on Coaching , 2017 .

[72]  Bastian Leibe,et al.  Robust Marker-Based Tracking for Measuring Crowd Dynamics , 2015, ICVS.

[73]  Leonid Sigal,et al.  Poselet Key-Framing: A Model for Human Activity Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[74]  D M Baer,et al.  Some current dimensions of applied behavior analysis. , 1968, Journal of applied behavior analysis.

[75]  Aubyn C. Stahmer,et al.  Naturalistic Developmental Behavioral Interventions: Empirically Validated Treatments for Autism Spectrum Disorder , 2015, Journal of Autism and Developmental Disorders.

[76]  R. Koegel,et al.  How To Teach Pivotal Behaviors to Children with Autism: A Training Manual. , 1988 .

[77]  L. Schreibman,et al.  Multiple peer use of pivotal response training to increase social behaviors of classmates with autism: results from trained and untrained peers. , 1997, Journal of applied behavior analysis.

[78]  L. Schreibman,et al.  Positive affect of parents of autistic children: A comparison across two teaching techniques* , 1991 .

[79]  R. Koegel,et al.  Producing speech use in nonverbal autistic children by reinforcing attempts , 1988, Journal of autism and developmental disorders.

[80]  L. Schreibman,et al.  Training parents to use the natural language paradigm to increase their autistic children's speech. , 1988, Journal of applied behavior analysis.

[81]  L. Koegel,et al.  Setting generalization of question-asking by children with autism. , 1998, American journal of mental retardation : AJMR.

[82]  Silvio Savarese,et al.  Structural-RNN: Deep Learning on Spatio-Temporal Graphs , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Manuel Giuliani,et al.  How can i help you': comparing engagement classification strategies for a robot bartender , 2013, ICMI '13.

[84]  Jessica Bradshaw,et al.  Assessing and Improving Early Social Engagement in Infants , 2014, Journal of positive behavior interventions.

[85]  Shrikanth S. Narayanan,et al.  Spoken dialog systems for children , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[86]  Aubyn C. Stahmer,et al.  Child Demographics Associated With Outcomes in a Community-Based Pivotal Response Training Program , 2007 .

[87]  Connie Kasari,et al.  Teacher-implemented joint attention intervention: pilot randomized controlled study for preschoolers with autism. , 2012, Journal of consulting and clinical psychology.

[88]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[89]  Lauren Ward,et al.  Improving Child Speech Disorder Assessment by Incorporating Out-of-Domain Adult Speech , 2017, INTERSPEECH.

[90]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[91]  Sanjit K. Mitra,et al.  Voice activity detection based on multiple statistical models , 2006, IEEE Transactions on Signal Processing.

[92]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Bayya Yegnanarayana,et al.  Single Frequency Filtering Approach for Discriminating Speech and Nonspeech , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[94]  Aubyn C. Stahmer,et al.  Dissemination of Evidence-Based Practice: Can We Train Therapists from a Distance? , 2009, Journal of autism and developmental disorders.

[95]  Sethuraman Panchanathan,et al.  Detecting Attention in Pivotal Response Treatment Video Probes , 2018, ICSM.

[96]  Russell Lang,et al.  Training Teachers to Assess the Challenging Behaviors of Students with Autism Using Video Tele-Conferencing , 2010 .

[97]  Jitendra Malik,et al.  Recurrent Network Models for Human Dynamics , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[98]  Meysam Asgari,et al.  Automatic analysis of pronunciations for children with speech sound disorders , 2018, Comput. Speech Lang..

[99]  A. Gulsrud,et al.  Randomized comparative efficacy study of parent-mediated interventions for toddlers with autism. , 2015, Journal of consulting and clinical psychology.

[100]  Dongxin Xu,et al.  Child vocalization composition as discriminant information for automatic autism detection , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[101]  L. Koegel,et al.  Language intervention and disruptive behavior in preschool children with autism , 1992, Journal of autism and developmental disorders.

[102]  J. Symon Expanding Interventions for Children With Autism , 2005 .

[103]  Hisashi Osumi,et al.  Modeling of Human Attention Based on Analysis of Magic , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[104]  Chang-Su Kim,et al.  Primary Object Segmentation in Videos Based on Region Augmentation and Reduction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[105]  Roland Göcke,et al.  Extending Long Short-Term Memory for Multi-View Structured Learning , 2016, ECCV.

[106]  Agata Rozga,et al.  Play with me — Measuring a child's engagement in a social interaction , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[107]  Stan Sclaroff,et al.  Learning Activity Progression in LSTMs for Activity Detection and Early Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[108]  John H. L. Hansen,et al.  Automatic assessment of language background in toddlers through phonotactic and pitch pattern modeling of short vocalizations , 2014, WOCCI.

[109]  L. Koegel,et al.  A Randomized Clinical Trial Comparison Between Pivotal Response Treatment (PRT) and Structured Applied Behavior Analysis (ABA) Intervention for Children with Autism , 2014, Journal of autism and developmental disorders.

[110]  S. Rogers,et al.  Telehealth for Expanding the Reach of Early Autism Training to Parents , 2012, Autism research and treatment.

[111]  Daniel Jurafsky,et al.  Shallow Semantic Parsing using Support Vector Machines , 2004, NAACL.

[112]  J. Bradshaw,et al.  Improving Question-Asking Initiations in Young Children with Autism Using Pivotal Response Treatment , 2013, Journal of Autism and Developmental Disorders.

[113]  Aubyn C. Stahmer,et al.  Classroom Pivotal Response Teaching for Children with Autism , 2011 .

[114]  Róbert Busa-Fekete,et al.  Determining Native Language and Deception Using Phonetic Features and Classifier Combination , 2016, INTERSPEECH.

[115]  Anastasia Kitsantas,et al.  Acquisition of Sport Knowledge and Skill , 2011 .

[116]  Luc Lecavalier,et al.  Moderators of Parent Training for Disruptive Behaviors in Young Children with Autism Spectrum Disorder , 2016, Journal of Abnormal Child Psychology.

[117]  Ying Zhang,et al.  Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks , 2016, INTERSPEECH.

[118]  Justin B. Leaf,et al.  Applied Behavior Analysis is a Science and, Therefore, Progressive , 2016, Journal of autism and developmental disorders.

[119]  G. Dawson,et al.  The Impact of Parent-Delivered Intervention on Parents of Very Young Children with Autism , 2014, Journal of autism and developmental disorders.

[120]  L. Schreibman,et al.  Collateral effects of parent training on family interactions , 1996, Journal of autism and developmental disorders.

[121]  Minsoo Hahn,et al.  Voice Activity Detection Using an Adaptive Context Attention Model , 2018, IEEE Signal Processing Letters.

[122]  Giuseppe De Pietro,et al.  A situation-aware system for the detection of motion disorders of patients with Autism Spectrum Disorders , 2014, Expert Syst. Appl..

[123]  Katarzyna Chawarska,et al.  Pivotal Response Treatment for Infants At-Risk for Autism Spectrum Disorders: A Pilot Study , 2012, Journal of Autism and Developmental Disorders.

[124]  Gwenn Englebienne,et al.  Towards Speech Emotion Recognition "in the Wild" Using Aggregated Corpora and Deep Multi-Task Learning , 2017, INTERSPEECH.

[125]  E. Jones,et al.  Parent Implemented Joint Attention Intervention for Preschoolers with Autism , 2007 .

[126]  Arsalane Zarghili,et al.  Stereotypical Motor Movement Recognition Using Microsoft Kinect with Artificial Neural Network , 2016 .

[127]  L. Koegel,et al.  A natural language teaching paradigm for nonverbal autistic children , 1987, Journal of autism and developmental disorders.

[128]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[129]  Juan Manuel Górriz,et al.  Hard C-means clustering for voice activity detection , 2006, Speech Commun..

[130]  Christophe Garcia,et al.  Visual Focus of Attention Estimation With Unsupervised Incremental Learning , 2016, IEEE Trans. Circuits Syst. Video Technol..

[131]  Tara N. Sainath,et al.  Large vocabulary automatic speech recognition for children , 2015, INTERSPEECH.

[132]  Patrick van der Smagt,et al.  Two-stream RNN/CNN for action recognition in 3D videos , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[133]  Truong Q. Nguyen,et al.  Context Matters: Refining Object Detection in Video with Recurrent Neural Networks , 2016, BMVC.