SuperpowerGlass

We have developed a system for automatic facial expression recognition running on Google Glass, delivering real-time social cues to children with Autism Spectrum Disorder (ASD). The system includes multiple mechanisms to engage children and their parents, who administer this technology within the home. We completed an at-home design trial with 14 families that used the learning aid over a 3-month period. We found that children with ASD generally respond well to wearing the system at home and opt for the most expressive feedback choice. We further evaluated app usage, facial engagement, and model accuracy. We found that the device can act as a powerful training aid when used periodically in the home, that interactive video content from wearable therapy sessions should be augmented with sufficient context about the content to produce long-term engagement, and that the design of wearable systems for children with ASD should be heavily dependent on the functioning level of the child. We contribute general design implications for developing wearable aids used by children with ASD and other behavioral disorders as well as their parents during at-home parent-administered therapy sessions.

[1]  Gillian R. Hayes,et al.  vSked: evaluation of a system to support classroom activities for children with autism , 2010, CHI.

[2]  Lizbeth Escobedo,et al.  Using Augmented Reality to Help Children with Autism Stay Focused , 2014, IEEE Pervasive Computing.

[3]  Daniel J. Faso,et al.  Context Effects on Facial Affect Recognition in Schizophrenia and Autism: Behavioral and Eye-Tracking Evidence. , 2016, Schizophrenia bulletin.

[4]  Brenda Smith Myles,et al.  Sensory Issues in Children with Asperger Syndrome and Autism , 2004 .

[5]  Marion Koelle,et al.  Don't look at me that way!: Understanding User Attitudes Towards Data Glasses Usage , 2015, MobileHCI.

[6]  Gillian R. Hayes,et al.  SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism , 2016, CHI.

[7]  Leah Findlater,et al.  Designing Conversation Cues on a Head-Worn Display to Support Persons with Aphasia , 2015, CHI.

[8]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[9]  Rosalind W. Picard,et al.  Affective Computing and Autism , 2006, Annals of the New York Academy of Sciences.

[10]  Matthew S. Goodwin,et al.  Automated Detection of Facial Expressions during Computer-Assisted Instruction in Individuals on the Autism Spectrum , 2017, CHI.

[11]  Fabio Pianesi,et al.  Enhancing social communication of children with high-functioning autism through a co-located interface , 2009, AI & SOCIETY.

[12]  Massimo Zancanaro,et al.  Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder , 2011, CHI.

[13]  M. Rutter,et al.  Adult outcome for children with autism. , 2004, Journal of child psychology and psychiatry, and allied disciplines.

[14]  Geraldine Dawson,et al.  Behavioral interventions in children and adolescents with autism spectrum disorder: a review of recent findings , 2011, Current opinion in pediatrics.

[15]  S. Baron-Cohen,et al.  Enhancing Emotion Recognition in Children with Autism Spectrum Conditions: An Intervention Using Animated Vehicles with Real Emotional Faces , 2010, Journal of autism and developmental disorders.

[16]  Terry Winograd,et al.  A practical approach to real-time neutral feature subtraction for facial expression recognition , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[17]  Katrina Jane Williams,et al.  Validity of the Social Communication Questionnaire in Assessing Risk of Autism in Preschool Children with Developmental Problems , 2007, Journal of autism and developmental disorders.

[18]  Bretagne Abirached,et al.  A framework for designing assistive technologies for teaching children with ASDs emotions , 2012, CHI Extended Abstracts.

[19]  Fabio Pianesi,et al.  Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD) , 2009, ITS '09.

[20]  Xinlong Jiang,et al.  ProCom: Designing and Evaluating a Mobile and Wearable System to Support Proximity Awareness for People with Autism , 2017, CHI.

[21]  Helen H. Epps,et al.  Color-emotion associations: Past experience and personal preference , 2004 .

[22]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[23]  Peter Washington,et al.  Superpower glass: delivering unobtrusive real-time social cues in wearable systems , 2016, UbiComp Adjunct.

[24]  Xiaoyang Tan,et al.  Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions , 2007, IEEE Transactions on Image Processing.

[25]  Peter Washington,et al.  A Wearable Social Interaction Aid for Children with Autism , 2016, CHI Extended Abstracts.

[26]  Amy L. Donaldson,et al.  Randomized, Controlled Trial of an Intervention for Toddlers With Autism: The Early Start Denver Model , 2010, Pediatrics.

[27]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[28]  Christopher Frauenberger,et al.  Participatory Evaluation with Autistic Children , 2017, CHI.

[29]  Gregory D. Abowd,et al.  Designing Capture Applications to Support the Education of Children with Autism , 2004, UbiComp.

[30]  Maja J. Mataric,et al.  Toward Socially Assistive Robotics for Augmenting Interventions for Children with Autism Spectrum Disorders , 2008, ISER.

[31]  Javier Hernandez,et al.  SenseGlass: using google glass to sense daily emotions , 2014, UIST.

[32]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[33]  Peter Kulchyski and , 2015 .

[34]  Lizbeth Escobedo,et al.  MOSOCO: a mobile assistive tool to support children with autism practicing social skills in real-life situations , 2012, CHI.

[35]  Christopher Frauenberger,et al.  Developing technology for autism: an interdisciplinary approach , 2011, Personal and Ubiquitous Computing.

[36]  Halley Profita,et al.  The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use , 2016, CHI.

[37]  L. Palumbo,et al.  Atypical emotional anticipation in high-functioning autism , 2015, Molecular Autism.

[38]  Toby Honsberger,et al.  The Effects of Covert Audio Coaching on Teaching Clerical Skills to Adolescents with Autism Spectrum Disorder , 2012, Journal of Autism and Developmental Disorders.

[39]  Jennifer C. Sarrett,et al.  Commentary: Attention to Eyes Is Present but in Decline in 2–6-Month-Old Infants Later Diagnosed with Autism , 2015, Front. Public Health.

[40]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[41]  O. I. Lovaas,et al.  Behavioral treatment and normal educational and intellectual functioning in young autistic children. , 1987, Journal of consulting and clinical psychology.

[42]  E. Stuart,et al.  Intervention targeting development of socially synchronous engagement in toddlers with autism spectrum disorder: a randomized controlled trial. , 2011, Journal of child psychology and psychiatry, and allied disciplines.

[43]  P. Ekman An argument for basic emotions , 1992 .

[44]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[45]  Christopher Frauenberger,et al.  When Empathy Is Not Enough: Assessing the Experiences of Autistic Children with Technologies , 2017, CHI.

[46]  B. Scassellati,et al.  Robots for use in autism research. , 2012, Annual review of biomedical engineering.

[47]  Gillian R. Hayes,et al.  Interactive visual supports for children with autism , 2010, Personal and Ubiquitous Computing.

[48]  Laurence Chaby,et al.  A Multidimensional Approach to the Study of Emotion Recognition in Autism Spectrum Disorders , 2015, Front. Psychol..

[49]  Yulei Fan,et al.  Collaborative Virtual Environment Technology for People With Autism , 2005, Fifth IEEE International Conference on Advanced Learning Technologies (ICALT'05).

[50]  G. Dawson,et al.  Understanding the Nature of Face Processing Impairment in Autism: Insights From Behavioral and Electrophysiological Studies , 2005, Developmental neuropsychology.

[51]  Christopher D. Manning,et al.  Hierarchical Bayesian Domain Adaptation , 2009, NAACL.

[52]  C. Boyatzis,et al.  Children's emotional associations with colors. , 1994, The Journal of genetic psychology.

[53]  Elizabeth Garrett-Mayer,et al.  Social and communication development in toddlers with early and later diagnosis of autism spectrum disorders. , 2007, Archives of general psychiatry.

[54]  Brian Reichow,et al.  Sensitivity and specificity of proposed DSM-5 diagnostic criteria for autism spectrum disorder. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[55]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[56]  Daniel C. Krawczyk,et al.  Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism , 2012, Journal of autism and developmental disorders.

[57]  Anind K. Dey,et al.  Parent-driven use of wearable cameras for autism support: a field study with families , 2012, UbiComp.

[58]  Alea Chandler Teeters,et al.  Use of a wearable camera system in conversation : toward a companion tool for social-emotional learning in autism , 2007 .

[59]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[60]  Gillian R. Hayes,et al.  vSked: an interactive visual schedule system for use in classrooms for children with autism , 2010, IDC.

[61]  Rosalind W. Picard Affective Computing: From Laughter to IEEE , 2010 .