Using Crowdsourcing to Train Facial Emotion Machine Learning Models with Ambiguous Labels

Peter Washington, Bioengineering, Stanford University, Stanford, California, United States Onur Cezmi Mutlu, Stanford University, Stanford, California, United States Emilie Leblanc, Pediatrics, Stanford University, Stanford, California, United States Aaron Kline, Pediatrics, Stanford University, Stanford, California, United States Cathy Hou, Computer Science, Stanford University, Stanford, California, United States Brianna Chrisman, Bioengineering, Stanford University, Stanford, California, United States Nate Stockham, Neuroscience, Stanford University, Stanford, California, United States Kelley Paskov, Biomedical Data Science, Stanford University, Stanford, California, United States Catalin Voss, Computer Science, Stanford University, Stanford, California, United States Nick Haber, Graduate School of Education, Stanford, Stanford, California, United States Dennis P. Wall, Pediatrics, Stanford University, Stanford, California, United States

[1]  Phuoc Tran-Gia,et al.  Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com , 2011, 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[2]  Eva Cerezo,et al.  Advanced Human Affect Visualization , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[3]  Jürgen S. Sauer,et al.  AniSAM & AniAvatar: Animated Visualizations of Affective States , 2016, CHI.

[4]  P. Ekman.  Are there basic emotions? , 1992, Psychological review.

[5]  Yordan Penev,et al.  Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition , 2020, Journal of personalized medicine.

[6]  S. White,et al.  Feasibility of Automated Training for Facial Emotion Expression and Recognition in Autism. , 2017, Behavior therapy.

[7]  Peter Washington,et al.  Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism , 2018, npj Digital Medicine.

[8]  Lori Lamel,et al.  Challenges in real-life emotion annotation and machine learning based detection , 2005, Neural Networks.

[9]  Catalin Voss,et al.  Making emotions transparent: Google Glass helps autistic kids understand facial expressions through augmented-reaiity therapy , 2020, IEEE Spectrum.

[10]  Vanessa Lobue,et al.  The Child Affective Facial Expression (CAFE) set: validity and reliability from untrained adults , 2014, Front. Psychol..

[11]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[12]  Todd Kulesza,et al.  Structured labeling for facilitating concept evolution in machine learning , 2014, CHI.

[13]  Peter Washington,et al.  Feasibility Testing of a Wearable Behavioral Aid for Social Learning in Children with Autism , 2018, Applied Clinical Informatics.

[14]  Dennis P. Wall,et al.  Multi-modular AI Approach to Streamline Autism Diagnosis in Young Children , 2020, Scientific Reports.

[15]  Heinrich Hußmann,et al.  Adjunct Proceedings of the 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications , 2018 .

[16]  Zheng Li,et al.  Implementing emotion-based user-aware e-learning , 2009, CHI Extended Abstracts.

[17]  Arshya Vahabzadeh,et al.  Feasibility of an Autism-Focused Augmented Reality Smartglasses System for Social Communication and Behavioral Coaching , 2017, Front. Pediatr..

[18]  Haik Kalantarian,et al.  Data-Driven Diagnostics and the Potential of Mobile Artificial Intelligence for Digital Therapeutic Phenotyping in Computational Psychiatry. , 2019, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  D. Wall,et al.  Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning , 2015, Translational Psychiatry.

[21]  Peter C. Wright,et al.  Empathy and experience in HCI , 2008, CHI.

[22]  Vanessa Lobue,et al.  Through the eyes of a child: preschoolers’ identification of emotional expressions from the child affective facial expression (CAFE) set , 2018, Cognition & emotion.

[23]  Daniel McDuff,et al.  Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected "In-the-Wild" , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[24]  R. Igliozzi,et al.  An android for enhancing social skills and emotion recognition in people with autism , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[25]  Catalin Voss,et al.  SUPERPOWER GLASS , 2019, GETMBL.

[26]  Karina Oertel,et al.  Emotions in HCI: an affective e-learning system , 2006 .

[27]  Todd F. DeLuca,et al.  Use of machine learning to shorten observation-based screening and diagnosis of autism , 2012, Translational Psychiatry.

[28]  Peter Washington,et al.  Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic Tasks , 2019, Journal of medical Internet research.

[29]  Ece Kamar,et al.  Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets , 2017, CHI.

[30]  D. Wall,et al.  Use of machine learning for behavioral distinction of autism and ADHD , 2016, Translational Psychiatry.

[31]  Peter Washington,et al.  Superpower glass: delivering unobtrusive real-time social cues in wearable systems , 2016, UbiComp Adjunct.

[32]  Detection of Emotional Events utilizing Support Vector Methods in an Active Learning HCI Scenario , 2014, ERM4HCI '14.

[33]  T. Dalgleish.  Basic Emotions , 2004 .

[34]  Martin Magdin,et al.  Real Time Facial Expression Recognition Using Webcam and SDK Affectiva , 2018, Int. J. Interact. Multim. Artif. Intell..

[35]  Ying Zhou,et al.  Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognition , 2020, ArXiv.

[36]  Peter Washington,et al.  Effect of Wearable Digital Intervention for Improving Socialization in Children With Autism Spectrum Disorder: A Randomized Clinical Trial , 2019, JAMA pediatrics.

[37]  Geraldine Fitzpatrick,et al.  Teaching and Developing Social and Emotional Skills with Technology , 2015, ACM Trans. Comput. Hum. Interact..

[38]  Maja J. Mataric,et al.  A Framework for Automatic Human Emotion Classification Using Emotion Profiles , 2011, IEEE Transactions on Audio, Speech, and Language Processing.

[39]  Peter Washington,et al.  Mobile detection of autism through machine learning on home video: A development and prospective validation study , 2018, PLoS medicine.

[40]  A. Prasad Vinod,et al.  Facial emotion recognition system for autistic children: a feasible study based on FPGA implementation , 2015, Medical & Biological Engineering & Computing.

[41]  Yusuke Ijima,et al.  Soft-Target Training with Ambiguous Emotional Utterances for DNN-Based Speech Emotion Classification , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  D. Wall,et al.  Clinical Evaluation of a Novel and Mobile Autism Risk Assessment , 2016, Journal of Autism and Developmental Disorders.

[45]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[46]  Peter Washington,et al.  A Wearable Social Interaction Aid for Children with Autism , 2016, CHI Extended Abstracts.

[47]  Jeffrey Heer,et al.  Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks , 2016, CSCW.

[48]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[49]  Daniel McDuff,et al.  AFFDEX SDK: A Cross-Platform Real-Time Multi-Face Expression Recognition Toolkit , 2016, CHI Extended Abstracts.

[50]  Petr Slovák,et al.  Designing Social and Emotional Skills Training: The Challenges and Opportunities for Technology Support , 2015, CHI.

[51]  Carlos Busso,et al.  Interpreting ambiguous emotional expressions , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[52]  Marian Stewart Bartlett,et al.  Emotion Mirror: A Novel Intervention for Autism Based on Real-Time Expression Recognition , 2012, ECCV Workshops.

[53]  Peter Washington,et al.  Detecting Developmental Delay and Autism Through Machine Learning Models Using Home Videos of Bangladeshi Children: Development and Validation Study , 2019, Journal of medical Internet research.

[54]  Yong Tao,et al.  Compound facial expressions of emotion , 2014, Proceedings of the National Academy of Sciences.

[55]  Yordan Penev,et al.  Feature replacement methods enable reliable home video analysis for machine learning detection of autism , 2020, Scientific Reports.

[56]  Haik Kalantarian,et al.  Feature Selection and Dimension Reduction of Social Autism Data , 2019, PSB.

[57]  Peter Washington,et al.  SuperpowerGlass , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[58]  D. Wall,et al.  Crowdsourced validation of a machine-learning classification system for autism and ADHD , 2017, Translational Psychiatry.