Superpower glass: delivering unobtrusive real-time social cues in wearable systems

We have developed a system for automatic facial expression recognition, which runs on Google Glass and delivers real-time social cues to the wearer. We evaluate the system as a behavioral aid for children with Autism Spectrum Disorder (ASD), who can greatly benefit from real-time non-invasive emotional cues and are more sensitive to sensory input than neurotypically developing children. In addition, we present a mobile application that enables users of the wearable aid to review their videos along with auto-curated emotional information on the video playback bar. This integrates our learning aid into the context of behavioral therapy. Expanding on our previous work describing in-lab trials, this paper presents our system and application-level design decisions in depth as well as the interface learnings gathered during the use of the system by multiple children with ASD in an at-home iterative trial.

[1]  Peter Washington,et al.  A Wearable Social Interaction Aid for Children with Autism , 2016, CHI Extended Abstracts.

[2]  Frank Dellaert,et al.  SWAN: System for Wearable Audio Navigation , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[3]  Fabio Pianesi,et al.  Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD) , 2009, ITS '09.

[4]  Laura Addison,et al.  A rapid assessment of skills in young children with autism. , 2004, Journal of applied behavior analysis.

[5]  Javier Hernandez,et al.  SenseGlass: using google glass to sense daily emotions , 2014, UIST.

[6]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[7]  Yung-Hsiang Lu,et al.  Cloud Computing for Mobile Users: Can Offloading Computation Save Energy? , 2010, Computer.

[8]  Maja J. Mataric,et al.  Toward Socially Assistive Robotics for Augmenting Interventions for Children with Autism Spectrum Disorders , 2008, ISER.

[9]  Arman Savran,et al.  Bosphorus Database for 3D Face Analysis , 2008, BIOID.

[10]  Nathan Sakunkoo.  GliFlix: Using Movie Subtitles for Language Learning , 2009 .

[11]  Alea Chandler Teeters,et al.  Use of a wearable camera system in conversation : toward a companion tool for social-emotional learning in autism , 2007 .

[12]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[13]  Paramvir Bahl,et al.  Energy characterization and optimization of image sensing toward continuous mobile vision , 2013, MobiSys '13.

[14]  Massimo Zancanaro,et al.  Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder , 2011, CHI.

[15]  Rosalind W. Picard.  Affective Computing for HCI , 1999, HCI.

[16]  P. Valdez,et al.  Effects of color on emotions. , 1994, Journal of experimental psychology. General.

[17]  Florian Alt,et al.  GravitySpot: Guiding Users in Front of Public Displays Using On-Screen Visual Cues , 2015, UIST.

[18]  Christine L. Lisetti.  Affective computing , 1998, Pattern Analysis and Applications.

[19]  Lijun Yin,et al.  A high-resolution 3D dynamic facial expression database , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[20]  Rosalind W. Picard.  Affective Computing , 1997 .

[21]  Michael A. Goodrich,et al.  A case for low-dose robotics in autism therapy , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[22]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[23]  Rosalind W. Picard.  Affective Computing: From Laughter to IEEE , 2010 .

[24]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[25]  Björn Hartmann,et al.  Video digests: a browsable, skimmable format for informational lecture videos , 2014, UIST.

[26]  Julie A. Kientz.  Designing and Developing Technology for Caregivers of Individuals with Autism , 2007 .

[27]  D Strickland,et al.  Virtual reality for the treatment of autism. , 1997, Studies in health technology and informatics.

[28]  Katsuhiko Sakaue,et al.  VizWear: Toward Human-Centered Interaction through Wearable Vision and Visualization , 2001, IEEE Pacific Rim Conference on Multimedia.

[29]  S. Baron-Cohen,et al.  Enhancing Emotion Recognition in Children with Autism Spectrum Conditions: An Intervention Using Animated Vehicles with Real Emotional Faces , 2010, Journal of autism and developmental disorders.

[30]  Terry Winograd,et al.  A practical approach to real-time neutral feature subtraction for facial expression recognition , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[31]  Fabio Pianesi,et al.  Enhancing social communication of children with high-functioning autism through a co-located interface , 2009, AI & SOCIETY.

[32]  Matthew S. Goodwin,et al.  Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder , 2008, Assets '08.

[33]  Sethuraman Panchanathan,et al.  VibroGlove: an assistive technology aid for conveying facial expressions , 2010, CHI Extended Abstracts.

[34]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[35]  D. Massaro,et al.  Development and Evaluation of a Computer-Animated Tutor for Vocabulary and Language Learning in Children with Autism , 2003, Journal of autism and developmental disorders.

[36]  Pattie Maes,et al.  eye-q: eyeglass peripheral display for subtle intimate notifications , 2006, Mobile HCI.

[37]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression (PIE) database , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[38]  Daniel C. Krawczyk,et al.  Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism , 2012, Journal of autism and developmental disorders.

[39]  Brenda Smith Myles,et al.  Sensory Issues in Children with Asperger Syndrome and Autism , 2004 .

[40]  Rosalind W. Picard,et al.  Affective Computing and Autism , 2006, Annals of the New York Academy of Sciences.

[41]  Xiaoyang Tan,et al.  Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions , 2007, IEEE Transactions on Image Processing.

[42]  Bretagne Abirached,et al.  A framework for designing assistive technologies for teaching children with ASDs emotions , 2012, CHI Extended Abstracts.

[43]  O. Hetzroni,et al.  Effects of a Computer-Based Intervention Program on the Communicative Functions of Children with Autism , 2004, Journal of autism and developmental disorders.

[44]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.