Automatic assessment of cognitive and emotional states in virtual reality-based flexibility training for four adolescents with autism

Tracking students? learning states to provide tailored learner support is a critical element of an adaptive learning system. This study explores how an automatic assessment is capable of tracking learners? cognitive and emotional states during virtual reality (VR)-based representational-flexibility training. This VR-based training program aims to promote the flexibility of adolescents with autism spectrum disorder (ASD) in interpreting, selecting and creating multimodal representations during STEM-related design problem solving. For the automatic assessment, we used both natural language processing (NLP) and machine-learning techniques to develop a multi-label classification model. We then trained the model with the data from a total of audio- and video-recorded 66 training sessions of four adolescents with ASD. To validate the model, we implemented both k-fold cross-validations and the manual evaluations by expert reviewers. The study finding suggests the feasibility of implementing the NLP and machine-learning driven automatic assessment to track and assess the cognitive and emotional states of individuals with ASD during VR-based flexibility training. The study finding also denotes the importance and viability of providing adaptive supports to maintain learners? cognitive and affective engagement in a highly interactive digital learning environment.

[1]  Gerald Kulm,et al.  Creativity and the Design of Music-Mathematics Activities in a Virtual Simulation Learning Environment , 2018 .

[2]  Martin H. Schmidt,et al.  Inhibition, flexibility, working memory and planning in autism spectrum disorders with and without comorbid ADHD-symptoms , 2008, Child and adolescent psychiatry and mental health.

[3]  Manolis Mavrikis,et al.  Affective learning: improving engagement and enhancing learning with affect-aware feedback , 2017, User Modeling and User-Adapted Interaction.

[4]  Jeroen Derks,et al.  Diversifying experiences enhance cognitive flexibility , 2012 .

[5]  M. O'Reilly,et al.  Assessing Behavioral Flexibility in Individuals With Developmental Disabilities , 2006 .

[6]  Agata Rozga,et al.  Automatic assessment of problem behavior in individuals with developmental disabilities , 2012, UbiComp.

[7]  H. Geurts,et al.  Working memory and cognitive flexibility-training for children with an autism spectrum disorder: a randomized controlled trial. , 2015, Journal of child psychology and psychiatry, and allied disciplines.

[8]  Anind K. Dey,et al.  Why do they still use paper?: understanding data collection and use in Autism education , 2013, CHI.

[9]  J. Stroop.  Studies of interference in serial verbal reactions. , 1992 .

[10]  Kim Van Naarden Braun,et al.  Development of a Machine Learning Algorithm for the Surveillance of Autism Spectrum Disorder , 2016, PloS one.

[11]  The Effect of Learning Experiences and Context on Infant Imitation and Generalization , 2008 .

[12]  B. Mcmahon,et al.  Adolescence to adulthood in STEM education and career development: The experience of students at the intersection of underrepresented minority status and disability , 2013 .

[13]  Xin Wei,et al.  Science, Technology, Engineering, and Mathematics (STEM) Participation Among College Students with an Autism Spectrum Disorder , 2012, Journal of Autism and Developmental Disorders.

[14]  Albert J. Cotugno,et al.  Social Competence and Social Skills Training and Intervention for Children with Autism Spectrum Disorders , 2009, Journal of autism and developmental disorders.

[15]  A. Batten.  Inclusion and the autism spectrum , 2005 .

[16]  J. Elen,et al.  Conceptualising, investigating and stimulating representational flexibility in mathematical problem solving and learning: a critical review , 2009 .

[17]  Jacqueline Ravet.  Delving deeper into the black box: formative assessment, inclusion and learners on the autism spectrum , 2013 .

[18]  P. Mitchell,et al.  The potential of virtual reality in social skills training for people with autistic spectrum disorders. , 2002, Journal of intellectual disability research : JIDR.

[19]  Laura K. Allen,et al.  Natural Language Processing and Learning Analytics , 2017 .

[20]  D. Naiman,et al.  Randomized controlled effectiveness trial of executive function intervention for children on the autism spectrum. , 2014, Journal of child psychology and psychiatry, and allied disciplines.

[21]  Robert T. Schultz,et al.  Automatic Detection of Autism Spectrum Disorder in Children Using Acoustic and Text Features from Brief Natural Conversations , 2019, INTERSPEECH.

[22]  Sean P. Yee,et al.  Developing a Contextualization of Students’ Mathematical Problem Solving , 2014 .

[23]  Johanna Pirker,et al.  MaroonVR - An Interactive and Immersive Virtual Reality Physics Laboratory , 2019, Learning in a Digital World.

[24]  P. Shekelle,et al.  Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments , 2012 .

[25]  H. Geurts,et al.  The paradox of cognitive flexibility in autism , 2009, Trends in Cognitive Sciences.

[26]  P. Reed,et al.  Cross-Modal Attention-Switching is Impaired in Autism Spectrum Disorders , 2011, Journal of Autism and Developmental Disorders.

[27]  Ji-Hyun Kim,et al.  Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , 2009, Comput. Stat. Data Anal..

[28]  A. Gagatsis,et al.  Representational Flexibility and Problem-Solving Ability in Fraction and Decimal Number Addition: A Structural Model , 2016 .

[29]  Diego Reforgiato Recupero,et al.  Bridging learning analytics and Cognitive Computing for Big Data classification in micro-learning video collections , 2019, Comput. Hum. Behav..

[30]  C. Clark,et al.  Underpinnings of the Costs of Flexibility in Preschool Children: The Roles of Inhibition and Working Memory , 2010, Developmental neuropsychology.

[31]  Tassos A. Mikropoulos,et al.  An Ensemble-Based Semi-Supervised Approach for Predicting Students’ Performance , 2018 .

[32]  Zane Stromberga,et al.  Virtualisation devices for student learning: Comparison between desktop-based (Oculus Rift) and mobile-based (Gear VR) virtual reality in medical and health science education , 2017 .

[33]  José Paulo Leal,et al.  EmoSpell, a Morphological and Emotional Word Analyzer , 2018, Inf..

[34]  Tetsuya Takiguchi,et al.  Detecting Abnormal Word Utterances in Children With Autism Spectrum Disorders , 2017, Perceptual and motor skills.

[35]  D. L. Linebarger,et al.  Scientific Concepts, Multiple Modalities, and Young Children , 2016 .

[36]  Fengfeng Ke,et al.  Virtual reality based collaborative design by children with high-functioning autism: design-based flexibility, identity, and norm construction , 2016, Interact. Learn. Environ..

[37]  Minsu Ha,et al.  Assessing Scientific Practices Using Machine-Learning Methods: How Closely Do They Match Clinical Interview Performance? , 2013, Journal of Science Education and Technology.

[38]  O. Pascalis,et al.  Visual recognition memory across contexts. , 2011, Developmental science.

[39]  Martin Mozina,et al.  Orange: data mining toolbox in python , 2013, J. Mach. Learn. Res..

[40]  R. Hastings,et al.  A population-based investigation of behavioural and emotional problems and maternal mental health: associations with autism spectrum disorder and intellectual disability. , 2011, Journal of child psychology and psychiatry, and allied disciplines.