The use of augmented reality in the diagnosis and treatment of autistic children: a review and a new system

This paper consists of two parts. The first presents a review of the literature on the use of augmented reality (AR) in the diagnosis and treatment of autistic children with a particular focus on the efficacy of AR in assisting autistic children who have communicative, social, sentiment, and attention deficit disorders. The review also investigated interactions between AR systems and children, taking into consideration the target behaviors that are selected from the child during treatment. Such modes were fully explored by taking into account the needs of the individual child in terms of achieving an improvement in their condition. Most significantly, the empirical information that was obtained from the reviewed works was evaluated according to some specific targeted attitudes and how each AR solution was utilized during treatment to achieve the fostering of such attitudes in order to identify the requirements for building an effective AR system. In addition, the review revealed the essential design features that can enable AR systems to achieve a high level of effectiveness in autism therapy. The review also covered the instruction that AR systems were noticed to execute, and focused on the significant characteristics that allow AR systems to accomplish degrees of efficacy in autism treatment. The review ends by classifying the various AR systems based on different criteria. The second part of the paper focuses on our new AR system as a case study. It covers the design considerations and decisions as well as the key features and appearance of the system. The paper concludes by making some recommendations for the further development of an AR system for application in the domain of child autism.

[1]  Philipp Michel,et al.  The Use of Technology in the Study , Diagnosis and Treatment of Autism , 2004 .

[2]  Chris Dede,et al.  Augmented Reality Teaching and Learning , 2014 .

[3]  Adel Al-Jumaily,et al.  Augmented Reality Game Therapy For Children With Autism Spectrum Disorder , 2017 .

[4]  Jason M. O'Kane,et al.  Adaptive Robot Design with Hand and Face Tracking for Use in Autism Therapy , 2010, ICSR.

[5]  K. Järbrink,et al.  The economic consequences of autistic spectrum disorder among children in a Swedish municipality , 2007, Autism : the international journal of research and practice.

[6]  Fotis Liarokapis,et al.  Using Augmented Reality as a Medium to Assist Teaching in Higher Education , 2010, Eurographics.

[7]  B. Scassellati,et al.  Robots for use in autism research. , 2012, Annual review of biomedical engineering.

[8]  Zhen Bai,et al.  Using Augmented Reality to Elicit Pretend Play for Children with Autism , 2015, IEEE Transactions on Visualization and Computer Graphics.

[9]  Chien-Hsu Chen,et al.  Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions , 2016, Comput. Hum. Behav..

[10]  Ru-Chu Shih,et al.  THE EFFECTS OF APPLYING GAME-BASED LEARNING TO WEBCAM MOTION SENSOR GAMES FOR AUTISTIC STUDENTS' SENSORY INTEGRATION TRAINING , 2012 .

[11]  Frank M. Gresham,et al.  The Relationship Between Adaptive Behavior and Social Skills , 1987 .

[12]  Phillip T. Pasqual,et al.  Using Technology in Autism Research: The Promise and the Perils , 2012, Computer.

[13]  Chien-Hsu Chen,et al.  Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. , 2015, Research in developmental disabilities.

[14]  Lizbeth Escobedo,et al.  SensoryPaint: a multimodal sensory intervention for children with neurodevelopmental disorders , 2014, UbiComp.

[15]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[16]  Lizbeth Escobedo,et al.  MOSOCO: a mobile assistive tool to support children with autism practicing social skills in real-life situations , 2012, CHI.

[17]  Adel Al-Jumaily,et al.  Early Diagnose of Autism Spectrum Disorder Using Machine Learning Based on Simple Upper Limb Movements , 2018, HIS.

[18]  Roger Lee Mendoza, PhD The Economics of Autism in Egypt , 2010 .

[19]  M J Weiss,et al.  Teaching Social Skills to People with Autism , 2001, Behavior modification.

[20]  Yee Mon Aung,et al.  SEMG based ANN for shoulder angle prediction , 2012 .

[21]  Ben Blachnitzky,et al.  Augmented reality , 2012, 2012 IEEE Hot Chips 24 Symposium (HCS).

[22]  Ernesto Damiani,et al.  Augmented reality technologies, systems and applications , 2010, Multimedia Tools and Applications.

[23]  N. S. Lakshmiprabha,et al.  [Poster] An augmented and virtual reality system for training autistic children , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[24]  C. Johnson,et al.  Identification and Evaluation of Children With Autism Spectrum Disorders , 2007, Pediatrics.

[25]  Kangdon Lee,et al.  Augmented Reality in Education and Training , 2012, TechTrends.

[26]  Arshya Vahabzadeh,et al.  Feasibility of an Autism-Focused Augmented Reality Smartglasses System for Social Communication and Behavioral Coaching , 2017, Front. Pediatr..

[27]  Khairul Anam,et al.  Evaluation of extreme learning machine for classification of individual and combined finger movements using electromyography on amputees and non-amputees , 2017, Neural Networks.

[28]  Pinata Winoto,et al.  Automatic Object Recognition in a Light-Weight Augmented Reality-based Vocabulary Learning Application for Children with Autism , 2019, ICIAI.

[29]  Emanuela Arvat,et al.  Neuroregulation of the Hypothalamus-Pituitary-Adrenal (HPA) Axis in Humans: Effects of GABA-, Mineralocorticoid-, and GH-Secretagogue-Receptor Modulation , 2006, TheScientificWorldJournal.

[30]  D. Baker,et al.  The Financial Side of Autism: Private and Public Costs , 2011 .

[31]  M. Knapp,et al.  Costs of autism spectrum disorders in the United Kingdom and the United States. , 2014, JAMA pediatrics.

[32]  Zhen Bai,et al.  Making Pretense Visible and Graspable: An augmented reality approach to promote pretend play , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[33]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[34]  Ebtehal Alsaggaf,et al.  Directions of Autism Diagnosis by Electroencephalogram Based brain Computer Interface : A Review , 2014 .

[35]  C. Dudley,et al.  The Value of Caregiver Time: Costs of Support and Care for Individuals Living with Autism Spectrum Disorder , 2014 .

[36]  M. Weinstein,et al.  Economic Burden of Childhood Autism Spectrum Disorders , 2012, Pediatrics.

[37]  Martin Knapp,et al.  Economic cost of autism in the UK , 2009, Autism : the international journal of research and practice.

[38]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[39]  Wendy E. Mackay,et al.  Augmented reality: linking real and virtual worlds: a new paradigm for interacting with computers , 1998, AVI '98.

[40]  Adel Al-Jumaily,et al.  An investigation of upper limb motor task based discriminate for high risk autism , 2017, 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE).

[41]  Jong-Il Park,et al.  Invisible marker tracking for AR , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[42]  Kerstin Dautenhahn,et al.  Collaborating with Kaspar: Using an autonomous humanoid robot to foster cooperative dyadic play among children with autism , 2010, 2010 10th IEEE-RAS International Conference on Humanoid Robots.

[43]  J. Prinz Which Emotions Are Basic , 2007 .

[44]  Adel Al-Jumaily,et al.  Upper limb motor coordination based early diagnosis in high risk subjects for Autism , 2016, 2016 IEEE Symposium Series on Computational Intelligence (SSCI).

[45]  Ramesh Raskar,et al.  Fundamentals: From Photons to Pixels , 2005 .

[46]  A. Meltzoff,et al.  Case Study of the Development of an Infant with Autism from Birth to Two Years of Age. , 2000, Journal of applied developmental psychology.

[47]  D. Reid,et al.  Virtual Reality in Pediatric Neurorehabilitation: Attention Deficit Hyperactivity Disorder, Autism and Cerebral Palsy , 2010, Neuroepidemiology.

[48]  Simon Baron-Cohen,et al.  Can we teach children with autism to understand emotions, belief, or pretence? , 1996, Development and Psychopathology.

[49]  Vladimir Geroimenko,et al.  Assisting Individuals with Autism and Cognitive Disorders: An Augmented Reality-Based Framework , 2019, Int. J. Online Biomed. Eng..

[50]  Don D. McMahon,et al.  Augmented Reality as a Navigation Tool to Employment Opportunities for Postsecondary Education Students With Intellectual Disabilities and Autism , 2015 .

[51]  日向 俊二 Kinect for Windowsアプリを作ろう , 2012 .

[52]  B. B. Zaidan,et al.  Multi-criteria analysis for OS-EMR software selection problem: A comparative study , 2015, Decis. Support Syst..

[53]  B. Garcia-Zapirain,et al.  Technologies as Support Tools for Persons with Autistic Spectrum Disorder: A Systematic Review , 2014, International journal of environmental research and public health.

[54]  Don D. McMahon,et al.  Augmented Reality for Teaching Science Vocabulary to Postsecondary Education Students With Intellectual Disabilities and Autism , 2016 .

[55]  Martin Knapp,et al.  The Economic Consequences of Autism in the UK , 2007 .

[56]  Lizbeth Escobedo,et al.  Object and Gesture Recognition to Assist Children with Autism during the Discrimination Training , 2012, CIARP.

[57]  Michael A. Goodrich,et al.  Detailed requirements for robots in autism therapy , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[58]  AkçayırMurat,et al.  Augmented reality in science laboratories , 2016 .

[59]  Karrie Karahalios,et al.  Vocsyl: visualizing syllable production for children with ASD and speech delays , 2010, ASSETS '10.

[60]  Jill Ashburner,et al.  Remote versus face-to-face delivery of early intervention programs for children with autism spectrum disorders: Perceptions of rural families and service providers , 2016 .

[61]  A. Billard,et al.  Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children With Autism , 2007, Assistive technology : the official journal of RESNA.

[62]  P. Markowitz Autism in a child with congenital cytomegalovirus infection , 1983, Journal of autism and developmental disorders.

[63]  William Gobson,et al.  Multimedia: From Wagner to Virtual Reality , 2001 .

[64]  H. S. S. Yee,et al.  Mobile technology for children with Autism Spectrum Disorder: Major trends and issues , 2012, 2012 IEEE Symposium on E-Learning, E-Management and E-Services.

[65]  Filomena Soares,et al.  An Augmented Reality GameBook for Children with Autism Spectrum Disorders , 2015 .

[66]  S. Spence,et al.  Autism from developmental and neuropsychological perspectives. , 2006, Annual review of clinical psychology.

[67]  Pamela J. Wolfberg,et al.  Including Children with Autism in Social and Imaginary Play with Typical Peers: Integrated Play Groups Model , 2012 .

[68]  Vassilios Morellas,et al.  A computer vision approach for the assessment of autism-related behavioral markers , 2012, 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL).

[69]  Antonio Cisternino,et al.  The FACE of autism , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[70]  Julius Verrel,et al.  Accuracy and Reliability of the Kinect Version 2 for Clinical Measurement of Motor Function , 2016, PloS one.

[71]  Mehmet Akif Ocak,et al.  Augmented reality in science laboratories: The effects of augmented reality on university students' laboratory skills and attitudes toward science laboratories , 2016, Comput. Hum. Behav..

[72]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[73]  Peter S Bearman,et al.  Socioeconomic Status and the Increased Prevalence of Autism in California , 2011, American sociological review.

[74]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[75]  Kim Adams,et al.  Using Lego robots to estimate cognitive ability in children who have severe physical disabilities , 2011, Disability and rehabilitation. Assistive technology.

[76]  Toolika Ghose,et al.  Thin is green: Leveraging the thin-client paradigm for sustainable mobile computing , 2015, Comput. Electr. Eng..

[77]  S. Marcus,et al.  Implications of Childhood Autism for Parental Employment and Earnings , 2012, Pediatrics.

[78]  Filomena Soares,et al.  Augmented reality for cognitive and social skills improvement in children with ASD , 2016, 2016 13th International Conference on Remote Engineering and Virtual Instrumentation (REV).

[79]  Christopher Gillberg,et al.  Early diagnosis of autism and impact on prognosis: a narrative review , 2013, Clinical epidemiology.

[80]  Satinder P. Singh,et al.  Introduction , 2002, British Journal of Ophthalmology.

[81]  Christoffer Pettersson,et al.  Challenges With Session to Session Managementin Brain Computer Interfaces : A Comparison of Classification Methods for Motor Imagery Induced EEG Patterns , 2014 .

[82]  Rod McCall,et al.  TimeWarp: interactive time travel with a mobile mixed reality game , 2008, Mobile HCI.

[83]  François Michaud,et al.  Investigating the Use of a Mobile Robotic Toy as an Imitation Agent for Children with Autism , 2006 .

[84]  Borko Furht,et al.  Augmented Reality: An Overview , 2011, Handbook of Augmented Reality.

[85]  Jessica Bradshaw,et al.  The importance of early identification and intervention for children with or at risk for autism spectrum disorders , 2014, International journal of speech-language pathology.

[86]  Jesús Favela,et al.  Living Labs for Pervasive Healthcare Research , 2015, IEEE Pervasive Computing.

[87]  S. Radhakrishnan Mobile augmented reality in modern libraries: Prospects and challenges , 2015 .

[88]  Vladimir Geroimenko Augmented Reality Technology and Art: The Analysis and Visualization of Evolving Conceptual Models , 2012, 2012 16th International Conference on Information Visualisation.

[89]  B. H. Lo,et al.  Autism Spectrum Disorder , 2018, Journal of paediatrics and child health.

[90]  Karola Dillenburger,et al.  Lost in Translation: Public Policies, Evidence-based Practice, and Autism Spectrum Disorder , 2014 .

[91]  Yu-Mei Huang,et al.  Augmented reality in educational activities for children with disabilities , 2016, Displays.

[92]  Diego López-de-Ipiña,et al.  TRIP: A Low-Cost Vision-Based Location System for Ubiquitous Computing , 2002, Personal and Ubiquitous Computing.

[93]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[94]  Peter S Bearman,et al.  Age of diagnosis for autism: individual and community factors across 10 birth cohorts , 2010, Journal of Epidemiology & Community Health.

[95]  Robert H. Riffenburgh,et al.  Linear Discriminant Analysis , 1960 .

[96]  Judith Kelner,et al.  Model Based Markerless 3D Tracking applied to Augmented Reality , 2010 .

[97]  Fabrizio Taffoni,et al.  A technological approach to studying motor planning ability in children at high risk for ASD , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[98]  Zhu-Hong You,et al.  Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis , 2013, BMC Bioinformatics.

[99]  Simon Baron-Cohen,et al.  Theory of mind in normal development and autism , 2001 .

[100]  Inmaculada Coma,et al.  A Kinect-based Augmented Reality System for Individuals with Autism Spectrum Disorders , 2018, GRAPP/IVAPP.

[101]  Jianmin Zheng,et al.  Design and Development of a Virtual Dolphinarium for Children With Autism , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[102]  J. Brian,et al.  Behavioral manifestations of autism in the first year of life , 2005, International Journal of Developmental Neuroscience.

[103]  Jyh-Chong Liang,et al.  Current status, opportunities and challenges of augmented reality in education , 2013, Comput. Educ..

[104]  P. Richard,et al.  Augmented Reality for Rehabilitation of Cognitive Disabled Children: A Preliminary Study , 2007, 2007 Virtual Rehabilitation.

[105]  Noel E. O'Connor,et al.  Classification of Sporting Activities Using Smartphone Accelerometers , 2013, Sensors.

[106]  Zhen Bai,et al.  Augmenting imagination for children with autism , 2012, IDC '12.

[107]  Woodrow Barfield,et al.  Augmented-reality displays , 1995 .

[108]  Anies Hannawati Purnamadjaja,et al.  Multi-Sensory synergies in Humanoid Robotics , 2004, Int. J. Humanoid Robotics.

[109]  Lizbeth Escobedo,et al.  A Smart Environment for Children with Autism , 2015, IEEE Pervasive Computing.

[110]  Adrian Iftene,et al.  Augmented Reality , 2010, 2010 12th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.

[111]  Livia Colle,et al.  Impaired recognition of negative basic emotions in autism: A test of the amygdala theory , 2006, Social neuroscience.

[112]  Don D. McMahon,et al.  Evaluating Augmented Reality to Complete a Chain Task for Elementary Students With Autism , 2016 .

[113]  F. Keller,et al.  Performance of Motor Sequences in Children at Heightened vs. Low Risk for ASD: A Longitudinal Study from 18 to 36 Months of Age , 2016, Front. Psychol..

[114]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[115]  Ivan E. Sutherland,et al.  The Ultimate Display , 1965 .

[116]  M. Mythili,et al.  A Study on Autism Spectrum Disorders using Classification Techniques , 2014 .

[117]  Ramesh Raskar,et al.  Modern approaches to augmented reality: introduction to current approaches , 2006, SIGGRAPH Courses.

[118]  Lizbeth Escobedo,et al.  Using Augmented Reality to Help Children with Autism Stay Focused , 2014, IEEE Pervasive Computing.

[119]  K. Scherer Toward a dynamic theory of emotion : The component process model of affective states , 1987 .

[120]  P. J. Krantz,et al.  Social interaction skills for children with autism: a script-fading procedure for beginning readers. , 1998, Journal of applied behavior analysis.

[121]  P. Olivier,et al.  Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease. , 2014, Gait & posture.

[122]  Maja J. Mataric,et al.  Graded cueing feedback in robot-mediated imitation practice for children with autism spectrum disorders , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[123]  William Stafford Noble,et al.  Support vector machine , 2013 .

[124]  Xiaoyong Liu,et al.  PSO-Based Support Vector Machine with Cuckoo Search Technique for Clinical Disease Diagnoses , 2014, TheScientificWorldJournal.

[125]  Sue Fletcher-Watson,et al.  A review of methods in the study of attention in autism , 2010 .

[126]  Frank Biocca,et al.  Comparing Differences in Presence during Social Interaction in Augmented Reality versus Virtual Reality Environments: An Exploratory Study , 2004 .

[127]  Mohammad O. Wedyan,et al.  Early diagnosis autism based on upper limb motor coordination in high risk subjects for autism , 2016, 2016 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS).

[128]  A. Ganapathiraju,et al.  LINEAR DISCRIMINANT ANALYSIS - A BRIEF TUTORIAL , 1995 .

[129]  Ana Grasielle Dionísio Corrêa,et al.  A Criatividade através da Expressão Musical: Uma Interface Gestual para Composição Musical Interativa , 2004 .

[130]  S. Janani,et al.  Augmented Reality Based Virtual Reality , 2012 .

[131]  E. Fombonne,et al.  Cost-impact of young adults with high-functioning autistic spectrum disorder. , 2007, Research in developmental disabilities.

[132]  Don D. McMahon,et al.  Effects of Digital Navigation Aids on Adults With Intellectual Disabilities , 2015 .

[133]  R. Philip,et al.  Emotion processing in autism spectrum disorder , 2009 .

[134]  Seung Yeon Choo,et al.  Interior Design in Augmented Reality Environment , 2010 .

[135]  P. Bearman,et al.  Diagnostic change and the increased prevalence of autism. , 2009, International journal of epidemiology.

[136]  A. Kai Qin,et al.  Evolutionary extreme learning machine , 2005, Pattern Recognit..

[137]  Steve Chi-Yin Yuen,et al.  Augmented Reality: An Overview and Five Directions for AR in Education , 2011 .

[138]  Morten Fjeld,et al.  Exploring Diminished Reality (DR) Spaces to Augment the Attention of Individuals with Autism , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops.

[139]  Rajiv Satsangi,et al.  The case for adopting virtual manipulatives in mathematics education for students with disabilities , 2017 .

[140]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[141]  Mehdi Mekni,et al.  Augmented Reality : Applications , Challenges and Future Trends , 2014 .

[142]  Tristan Fletcher,et al.  Support Vector Machines Explained , 2008 .

[143]  Gonzalo Lorenzo,et al.  Preliminary study of augmented reality as an instrument for improvement of social skills in children with autism spectrum disorder , 2018, Education and Information Technologies.

[144]  LDA (Linear Discriminant Analysis) , 2009, Encyclopedia of Biometrics.

[145]  Minho Lee,et al.  Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection , 2017, Neural Networks.

[146]  Olivier Nannipieri,et al.  New Augmented Reality Taxonomy: Technologies and Features of Augmented Environment , 2011, Handbook of Augmented Reality.

[147]  M. Ganz,et al.  The lifetime distribution of the incremental societal costs of autism. , 2007, Archives of pediatrics & adolescent medicine.

[148]  Shih-Hao Hung,et al.  Real-time and intelligent private data protection for the Android platform , 2015, Pervasive Mob. Comput..

[149]  Paolo Perego,et al.  Use of Machine Learning to Identify Children with Autism and Their Motor Abnormalities , 2015, Journal of autism and developmental disorders.

[150]  N. Sarat Chandra Babu,et al.  Assistive Learning for Children with Autism Using Augmented Reality , 2013, 2013 IEEE Fifth International Conference on Technology for Education (t4e 2013).

[151]  Hassan A. Karimi Introduction to Navigation , 2011 .

[152]  Sampathkumar Veeraraghavan,et al.  Exploration of Autism Expert Systems , 2007, Fourth International Conference on Information Technology (ITNG'07).

[153]  S. Groux,et al.  Pictogram Room: Natural Interaction Technologies to Aid in the Development of Children with Autism , 2014 .

[154]  Arshya Vahabzadeh,et al.  Safety and Lack of Negative Effects of Wearable Augmented-Reality Social Communication Aid for Children and Adults with Autism , 2017, bioRxiv.

[155]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[156]  Muhammad Mostafa Monowar,et al.  Rise of Augmented Reality: Current and Future Application Areas , 2013 .

[157]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.