Social-Emotional-Sensory Design Map for Affective Computing Informed by Neurodivergent Experiences

One of the grand challenges of artificial intelligence and affective computing is for technology to become emotionally-aware and thus, more human-like. Modeling human emotions is particularly complicated when we consider the lived experiences of people who are on the autism spectrum. To understand the emotional experiences of autistic adults and their attitudes towards common representations of emotions, we deployed a context study as the first phase of a Grounded Design research project. Based on community observations and interviews, this work contributes empirical evidence of how the emotional experiences of autistic adults are entangled with social interactions as well as the processing of sensory inputs. We learned that (1) the emotional experiences of autistic adults are embodied and co-constructed within the context of physical environments, social relationships, and technology use, and (2) conventional approaches to visually representing emotion in affective education and computing systems fail to accurately represent the experiences and perceptions of autistic adults. We contribute a social-emotional-sensory design map to guide designers in creating more diverse and nuanced affective computing interfaces that are enriched by accounting for neurodivergent users.

[1]  G. Baird,et al.  Alexithymia in Adolescents with Autism Spectrum Disorder: Its Relationship to Internalising Difficulties, Sensory Modulation and Social Cognition , 2015, Journal of Autism and Developmental Disorders.

[2]  Barbara Solenthaler,et al.  Affective State Prediction Based on Semi-Supervised Learning from Smartphone Touch Data , 2020, CHI.

[3]  Rosalind W. Picard,et al.  Affective Computing and Autism , 2006, Annals of the New York Academy of Sciences.

[4]  Jeffrey T. Hancock,et al.  Experimental evidence of massive-scale emotional contagion through social networks , 2014, Proceedings of the National Academy of Sciences.

[5]  Elizabeth B. Torres,et al.  Noise from the periphery in autism , 2013, Front. Integr. Neurosci..

[6]  Myounghoon Jeon,et al.  Emotions in Driving , 2017 .

[7]  Changchun Liu,et al.  Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder , 2008, Int. J. Hum. Comput. Stud..

[8]  Gillian R. Hayes,et al.  Making in Minecraft: A Means of Self-Expression for Youth with Autism , 2017, IDC.

[9]  O. Grynszpan,et al.  Innovative technology-based interventions for autism spectrum disorders: A meta-analysis , 2014, Autism : the international journal of research and practice.

[10]  Phoebe Sengers,et al.  The disenchantment of affect , 2008, Personal and Ubiquitous Computing.

[11]  T. Dalgleish Basic Emotions , 2004 .

[12]  Christopher Frauenberger,et al.  Agency of Autistic Children in Technology Research—A Critical Literature Review , 2019, ACM Trans. Comput. Hum. Interact..

[13]  R. Weissberg,et al.  Social and emotional learning: a framework for promoting mental health and reducing risk behavior in children and youth. , 2000, The Journal of school health.

[14]  J. Russell,et al.  The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology , 2005, Development and Psychopathology.

[15]  P. Reed,et al.  Children with autism spectrum disorder: teaching conversation involving feelings about events , 2017, Journal of intellectual disability research : JIDR.

[16]  Kathryn E. Ringland A Place to Play: The (Dis)Abled Embodied Experience for Autistic Children in Online Spaces , 2019, CHI.

[17]  Jessica K. Hodgins,et al.  Investigating the Influence of Avatar Facial Characteristics on the Social Behaviors of Children with Autism , 2016, CHI.

[18]  Donald A. Schön Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions , 1987 .

[19]  Sean A. Munson,et al.  Design Opportunities for Mental Health Peer Support Technologies , 2017, CSCW.

[20]  L. F. Barrett How Emotions Are Made: The Secret Life of the Brain , 2017 .

[21]  Daniel McDuff,et al.  AffectAura: an intelligent system for emotional memory , 2012, CHI.

[22]  K. Goerlich The Multifaceted Nature of Alexithymia – A Neuroscientific Perspective , 2018, Front. Psychol..

[23]  Nancy Longnecker,et al.  Doctor-patient communication: a review. , 2010, The Ochsner journal.

[24]  Wijnand A. IJsselsteijn,et al.  Caring for Vincent: A Chatbot for Self-Compassion , 2019, CHI.

[25]  M. Mazurek,et al.  Sensory over-responsivity, repetitive behavior, and emotional functioning in boys with and without autism spectrum disorder , 2020 .

[26]  A. Toga,et al.  Emotion Regulation , 2015 .

[27]  E. Cherland The Development of Emotional Competence , 2004 .

[28]  Peter Washington,et al.  SuperpowerGlass , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[29]  John McDonald,et al.  Sensory Integration and the Perceptual Experience of Persons with Autism , 2006, Journal of autism and developmental disorders.

[30]  R. Morris,et al.  Towards an Artificially Empathic Conversational Agent for Mental Health Applications: System Design and User Perceptions , 2018, Journal of medical Internet research.

[31]  Gillian R. Hayes,et al.  Interactive visual supports for children with autism , 2010, Personal and Ubiquitous Computing.

[32]  Stephen Voida,et al.  Visually Encoding the Lived Experience of Bipolar Disorder , 2019, CHI.

[33]  Chih-Hung Wu,et al.  Review of affective computing in education/learning: Trends and challenges , 2016, Br. J. Educ. Technol..

[34]  Jock D. Mackinlay,et al.  The design space of input devices , 1990, CHI '90.

[35]  C. Kerns,et al.  Traumatic Childhood Events and Autism Spectrum Disorder , 2015, Journal of autism and developmental disorders.

[36]  L. Ellingson Embodiment in Qualitative Research , 2017 .

[37]  Gregory D. Abowd,et al.  SocialMirror: motivating young adults with autism to practice life skills in a social world , 2012, CSCW.

[38]  J. Russell Core affect and the psychological construction of emotion. , 2003, Psychological review.

[39]  Karrie Karahalios,et al.  Skin Conductance as an InSitu Marker for Emotional Arousal in Children with Neurodevelopmental Communication Impairments: Methodological Considerations and Clinical Implications , 2022 .

[40]  J. Russell A circumplex model of affect. , 1980 .

[41]  Jon Whittle,et al.  Prototyping 'clasp': implications for designing digital technology for and with adults with autism , 2014, Conference on Designing Interactive Systems.

[42]  Gillian R. Hayes,et al.  SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism , 2016, CHI.

[43]  Mark Matthews,et al.  MoodLight: Exploring Personal and Social Implications of Ambient Display of Biosensor Data , 2015, CSCW.

[44]  Jennifer Ann Rode,et al.  Does Technology Have Race? , 2016, CHI Extended Abstracts.

[45]  David Peebles,et al.  Sorting Preference in Children with Autism: The Dominance of Concrete Features , 2007, Journal of autism and developmental disorders.

[46]  C. Gillberg,et al.  Children with autism spectrum disorders and selective mutism , 2018, Neuropsychiatric disease and treatment.

[47]  Sandrine Balbo,et al.  Towards co-design with users who have autism spectrum disorders , 2009, Universal Access in the Information Society.

[48]  J. Weiss,et al.  Bullying Experiences Among Children and Youth with Autism Spectrum Disorders , 2012, Journal of autism and developmental disorders.

[49]  Reed Stevens,et al.  Mapping the daily media round: novel methods for understanding families’ mobile technology use , 2018 .

[50]  Nazanin Andalibi,et al.  The Human in Emotion Recognition on Social Media: Attitudes, Outcomes, Risks , 2020, CHI.

[51]  Johnny Saldaña,et al.  The Coding Manual for Qualitative Researchers , 2009 .

[52]  Klaus Fiedler,et al.  Affect and Cognitive Processes in Educational Contexts , 2014 .

[53]  Volker Wulf,et al.  Socio-Informatics: A Practice-Based Perspective on the Design and Use of IT Artifacts , 2018 .

[54]  J. Hietanen,et al.  Bodily maps of emotions , 2013, Proceedings of the National Academy of Sciences.

[55]  Hilary Johnson,et al.  Diversity for design: a framework for involving neurodiverse children in the technology design process , 2014, CHI.

[56]  M Mary Konstantareas,et al.  Affect Regulation and Temperament in Children with Autism Spectrum Disorder , 2006, Journal of autism and developmental disorders.

[57]  Meryl Alper,et al.  Inclusive sensory ethnography: Studying new media and neurodiversity in everyday life , 2018, New Media Soc..

[58]  Paul Dourish,et al.  Implications for design , 2006, CHI.

[59]  D. Caughey Doctor:patient communication. , 1987, The New Zealand medical journal.

[60]  Micah O. Mazurek,et al.  Video Game Use and Problem Behaviors in Boys with Autism Spectrum Disorders. , 2013 .

[61]  D. W. Hoover The Effects of Psychological Trauma on Children with Autism Spectrum Disorders: a Research Review , 2015, Review Journal of Autism and Developmental Disorders.

[62]  G. Vaiva,et al.  Distress, Suicidality, and Affective Disorders at the Time of Social Networks , 2019, Current Psychiatry Reports.

[63]  Antonia Hamilton,et al.  Recognition of Emotions in Autism: A Formal Meta-Analysis , 2013, Journal of autism and developmental disorders.

[64]  H. D. Jaegher,et al.  Embodiment and sense-making in autism , 2013, Front. Integr. Neurosci..

[65]  Karin Barron Ethics in qualitative social research on marginalized groups , 1999 .

[66]  Dora M Raymaker,et al.  Collaboration Strategies in Nontraditional Community-Based Participatory Research Partnerships: Lessons From an Academic–Community Partnership With Autistic Self-Advocates , 2011, Progress in community health partnerships : research, education, and action.

[67]  Gillian R. Hayes,et al.  Mobile video modeling for employment interviews for individuals with autism , 2015 .

[68]  Helena M. Mentis,et al.  Concealment of Emotion in an Emergency Room: Expanding Design for Emotion Awareness , 2012, Computer Supported Cooperative Work (CSCW).

[69]  Matthew S. Goodwin,et al.  Sensor-enabled detection of stereotypical motor movements in persons with autism spectrum disorder , 2008, IDC.

[70]  Jean-Claude Martin,et al.  Human computer interfaces for autism: assessing the influence of task assignment and output modalities , 2005, CHI Extended Abstracts.

[71]  S. Baron-Cohen,et al.  Talent in autism: hyper-systemizing, hyper-attention to detail and sensory hypersensitivity , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[72]  Eric Horvitz,et al.  Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems , 2017, 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO).

[73]  Jennifer B. Ganz,et al.  The Use of Visual Supports to Facilitate Transitions of Students with Autism , 2000 .

[74]  S. Gutstein,et al.  Quality of Life for People with Autism: Raising the Standard for Evaluating Successful Outcomes. , 2007, Child and adolescent mental health.

[75]  Robert E. Kraut,et al.  Social use of computer-mediated communication by adults on the autism spectrum , 2010, CSCW '10.

[76]  R. Schaaf,et al.  Occupational therapy and sensory integration for children with autism: a feasibility, safety, acceptability and fidelity study , 2012, Autism : the international journal of research and practice.

[77]  Jaime Teevan,et al.  Optimizing for Happiness and Productivity: Modeling Opportune Moments for Transitions and Breaks at Work , 2020, CHI.

[78]  Gillian R. Hayes,et al.  Would You Be Mine: Appropriating Minecraft as an Assistive Technology for Youth with Autism , 2016, ASSETS.