Data-Driven Diagnostics and the Potential of Mobile Artificial Intelligence for Digital Therapeutic Phenotyping in Computational Psychiatry.

Data science and digital technologies have the potential to transform diagnostic classification. Digital technologies enable the collection of big data, and advances in machine learning and artificial intelligence enable scalable, rapid, and automated classification of medical conditions. In this review, we summarize and categorize various data-driven methods for diagnostic classification. In particular, we focus on autism as an example of a challenging disorder due to its highly heterogeneous nature. We begin by describing the frontier of data science methods for the neuropsychiatry of autism. We discuss early signs of autism as defined by existing pen-and-paper-based diagnostic instruments and describe data-driven feature selection techniques for determining the behaviors that are most salient for distinguishing children with autism from neurologically typical children. We then describe data-driven detection techniques, particularly computer vision and eye tracking, that provide a means of quantifying behavioral differences between cases and controls. We also describe methods of preserving the privacy of collected videos and prior efforts of incorporating humans in the diagnostic loop. Finally, we summarize existing digital therapeutic interventions that allow for data capture and longitudinal outcome tracking as the diagnosis moves along a positive trajectory. Digital phenotyping of autism is paving the way for quantitative psychiatry more broadly and will set the stage for more scalable, accessible, and precise diagnostic techniques in the field.

[1]  James W Tanaka,et al.  Using computerized games to teach face recognition skills to children with autism spectrum disorder: the Let's Face It! program. , 2010, Journal of child psychology and psychiatry, and allied disciplines.

[2]  Fadi Thabtah,et al.  An accessible and efficient autism screening method for behavioural data and predictive analyses , 2018, Health Informatics J..

[3]  Daniel P. Kennedy,et al.  Atypical Visual Saliency in Autism Spectrum Disorder Quantified through Model-Based Eye Tracking , 2015, Neuron.

[4]  A. Klin,et al.  Absence of preferential looking to the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder. , 2008, Archives of general psychiatry.

[5]  Peter Washington,et al.  Superpower glass: delivering unobtrusive real-time social cues in wearable systems , 2016, UbiComp Adjunct.

[6]  Pierluigi Carcagnì,et al.  Computational Assessment of Facial Expression Production in ASD Children , 2018, Sensors.

[7]  Andrew M. Michael,et al.  A Comparison of Structural Brain Imaging Findings in Autism Spectrum Disorder and Attention-Deficit Hyperactivity Disorder , 2015, Neuropsychology Review.

[8]  Matthew J. Huentelman,et al.  The Autism Genome Project , 2005, American journal of pharmacogenomics : genomics-related research in drug development and clinical practice.

[9]  Micol Spitale,et al.  Emotify: emotional game for children with autism spectrum disorder based-on machine learning , 2019, IUI Companion.

[10]  Matthew J. McAuliffe,et al.  Sharing Heterogeneous Data: The National Database for Autism Research , 2012, Neuroinformatics.

[11]  Peter Washington,et al.  Detecting Developmental Delay and Autism Through Machine Learning Models Using Home Videos of Bangladeshi Children: Development and Validation Study , 2019, Journal of medical Internet research.

[12]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[13]  Shaun J. Canavan,et al.  Combining gaze and demographic feature descriptors for autism classification , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[14]  Björn W. Schuller,et al.  CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[15]  Timothy A Keller,et al.  The nature of brain dysfunction in autism: functional brain imaging studies , 2010, Current opinion in neurology.

[16]  Bappaditya Mandal,et al.  Towards Automatic Screening of Typical and Atypical Behaviors in Children With Autism , 2019, 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[17]  Thaddeus Tarpey,et al.  Smartphone measures of day-to-day behavior changes in children with autism , 2018, npj Digital Medicine.

[18]  A. Klin,et al.  Attention to Eyes is Present But in Decline in 2–6 Month-Olds Later Diagnosed with Autism , 2013, Nature.

[19]  Stephen J. Blumberg,et al.  The Prevalence of Parent-Reported Autism Spectrum Disorder Among US Children , 2018, Pediatrics.

[20]  S. Baron-Cohen,et al.  The Autism-Spectrum Quotient (AQ): Evidence from Asperger Syndrome/High-Functioning Autism, Malesand Females, Scientists and Mathematicians , 2001, Journal of autism and developmental disorders.

[21]  Jianbo Shi,et al.  Social saliency prediction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Avid Roman-Gonzalez,et al.  Developing an eye-tracking algorithm as a potential tool for early diagnosis of autism spectrum disorder in children , 2017, PloS one.

[23]  F. Shic,et al.  Decreased Spontaneous Attention to Social Scenes in 6-Month-Old Infants Later Diagnosed with Autism Spectrum Disorders , 2013, Biological Psychiatry.

[24]  Russ B. Altman,et al.  An ontology for Autism Spectrum Disorder (ASD) to infer ASD phenotypes from Autism Diagnostic Interview-Revised data , 2015, J. Biomed. Informatics.

[25]  Nilanjan Sarkar,et al.  A Data-Driven Mobile Application for Efficient, Engaging, and Accurate Screening of ASD in Toddlers , 2018, HCI.

[26]  S. Bölte,et al.  ‘Emotiplay’: a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation , 2017, European Child & Adolescent Psychiatry.

[27]  A. Guastella,et al.  The Immune System, Cytokines, and Biomarkers in Autism Spectrum Disorder , 2017, Neuroscience Bulletin.

[28]  Jake K. Aggarwal,et al.  Spontaneous facial expression recognition: A robust metric learning approach , 2014, Pattern Recognit..

[29]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  S. Spence,et al.  The autism genetic resource exchange: a resource for the study of autism and related neuropsychiatric conditions. , 2001, American journal of human genetics.

[31]  Matthew S. Goodwin,et al.  Caregiver Daily Reporting of Symptoms in Autism Spectrum Disorder: Observational Study Using Web and Mobile Apps , 2022 .

[32]  Guillermo Sapiro,et al.  Computer Vision Analysis for Quantification of Autism Risk Behaviors , 2018, IEEE Transactions on Affective Computing.

[33]  Li Yi,et al.  Identifying children with autism spectrum disorder based on their face processing abnormality: A machine learning framework , 2016, Autism research : official journal of the International Society for Autism Research.

[34]  Matthew S. Goodwin,et al.  Applying Machine Learning to Facilitate Autism Diagnostics: Pitfalls and Promises , 2014, Journal of Autism and Developmental Disorders.

[35]  Daniel H. Geschwind,et al.  Genetics of autism spectrum disorders , 2011, Trends in Cognitive Sciences.

[36]  João Magalhães,et al.  Crowdsourcing facial expressions for affective-interaction , 2016, Comput. Vis. Image Underst..

[37]  Wendy Roberts,et al.  Structure of the autism symptom phenotype: A proposed multidimensional model. , 2007, Journal of the American Academy of Child and Adolescent Psychiatry.

[38]  Cynthia Breazeal,et al.  Personalized Estimation of Engagement From Videos Using Active Learning With Deep Reinforcement Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Katarzyna Chawarska,et al.  Gaze Response to Dyadic Bids at 2 Years Related to Outcomes at 3 Years in Autism Spectrum Disorders: A Subtyping Analysis , 2013, Journal of Autism and Developmental Disorders.

[40]  S. Baron-Cohen,et al.  The Q-CHAT (Quantitative CHecklist for Autism in Toddlers): A Normally Distributed Quantitative Measure of Autistic Traits at 18–24 Months of Age: Preliminary Report , 2008, Journal of autism and developmental disorders.

[41]  Dennis P. Wall,et al.  Machine learning approach for early detection of autism by combining questionnaire and home video screening , 2017, J. Am. Medical Informatics Assoc..

[42]  Vassilios Morellas,et al.  Computer Vision Tools for Low-Cost and Noninvasive Measurement of Autism-Related Behaviors in Infants , 2014, Autism research and treatment.

[43]  Daniel P. Kennedy,et al.  The Autism Brain Imaging Data Exchange: Towards Large-Scale Evaluation of the Intrinsic Brain Architecture in Autism , 2013, Molecular Psychiatry.

[44]  Marian Stewart Bartlett,et al.  Emotion Mirror: A Novel Intervention for Autism Based on Real-Time Expression Recognition , 2012, ECCV Workshops.

[45]  D. Wall,et al.  Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning , 2015, Translational Psychiatry.

[46]  Joseph Piven,et al.  Abnormal Use of Facial Information in High-Functioning Autism , 2007, Journal of autism and developmental disorders.

[47]  Terry Winograd,et al.  A practical approach to real-time neutral feature subtraction for facial expression recognition , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[48]  Mark A. Musen,et al.  The protégé project: a look back and a look forward , 2015, SIGAI.

[49]  James M. Rehg,et al.  Detecting eye contact using wearable eye-tracking glasses , 2012, UbiComp.

[50]  C. Lord,et al.  Austism diagnostic observation schedule: A standardized observation of communicative and social behavior , 1989, Journal of autism and developmental disorders.

[51]  Matthew S. Goodwin,et al.  Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder , 2018, Journal of Autism and Developmental Disorders.

[52]  Haik Kalantarian,et al.  Feature Selection and Dimension Reduction of Social Autism Data , 2019, PSB.

[53]  Kun Zhang,et al.  A pilot study on evaluating children with autism spectrum disorder using computer games , 2019, Comput. Hum. Behav..

[54]  A. Couteur,et al.  Autism Diagnostic Interview-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders , 1994, Journal of autism and developmental disorders.

[55]  A. Klin,et al.  Parsing heterogeneity in autism spectrum disorders: visual scanning of dynamic social scenes in school-aged children. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[56]  Gillian R. Hayes,et al.  vSked: evaluation of a system to support classroom activities for children with autism , 2010, CHI.

[57]  Henrik Eriksson,et al.  The evolution of Protégé: an environment for knowledge-based systems development , 2003, Int. J. Hum. Comput. Stud..

[58]  Björn W. Schuller,et al.  Personalized machine learning for robot perception of affect and engagement in autism therapy , 2018, Science Robotics.

[59]  Peter Washington,et al.  A Mobile Game for Automatic Emotion-Labeling of Images , 2020, IEEE Transactions on Games.

[60]  S. Rogers,et al.  Brief report: Early intervention in autism , 1996, Journal of autism and developmental disorders.

[61]  Matthew S. Goodwin,et al.  Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder , 2008, Assets '08.

[62]  Peter Washington,et al.  Labeling images with facial emotion and the potential for pediatric healthcare , 2019, Artif. Intell. Medicine.

[63]  James M. Rehg,et al.  Decoding Children's Social Behavior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[64]  Dennis P. Wall,et al.  The Quantified Brain: A Framework for Mobile Device-Based Assessment of Behavior and Neurological Function , 2016, Applied Clinical Informatics.

[65]  R. Anney,et al.  Autism genetics: opportunities and challenges for clinical translation , 2017, Nature Reviews Genetics.

[66]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[67]  Hidenori Yamasue,et al.  Computer-analyzed facial expression as a surrogate marker for autism spectrum social core symptoms , 2018, PloS one.

[68]  J. Piven,et al.  Visual Scanning of Faces in Autism , 2002, Journal of autism and developmental disorders.

[69]  Peter Washington,et al.  Mobile detection of autism through machine learning on home video: A development and prospective validation study , 2018, PLoS medicine.

[70]  Kiely N. James,et al.  Autism risk in offspring can be assessed through quantification of male sperm mosaicism , 2019, Nature Medicine.

[71]  Warren Jones,et al.  Mechanisms of Diminished Attention to Eyes in Autism. , 2017, The American journal of psychiatry.

[72]  F. Shic,et al.  Computer-Assisted Face Processing Instruction Improves Emotion Recognition, Mentalizing, and Social Skills in Students with ASD , 2015, Journal of Autism and Developmental Disorders.

[73]  Ugis Sarkans,et al.  Biomarkers in autism spectrum disorder: the old and the new , 2014, Psychopharmacology.

[74]  Agnieszka Landowska,et al.  Automatic recognition of therapy progress among children with autism , 2017, Scientific Reports.

[75]  C. Lord,et al.  The Simons Simplex Collection: A Resource for Identification of Autism Genetic Risk Factors , 2010, Neuron.

[76]  Guillermo Sapiro,et al.  Automatic emotion and attention analysis of young children at home: a ResearchKit autism feasibility study , 2018, npj Digital Medicine.

[77]  Gondy Leroy,et al.  Smartphones to facilitate communication and improve social skills of children with severe autism spectrum disorder: special education teachers as proxies , 2008, IDC.

[78]  C. Schaaf,et al.  Autism genetics – an overview , 2017, Prenatal diagnosis.

[79]  Erik Linstead,et al.  Applications of Supervised Machine Learning in Autism Spectrum Disorder Research: a Review , 2019, Review Journal of Autism and Developmental Disorders.

[80]  Peter Washington,et al.  Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic Tasks , 2019, Journal of medical Internet research.

[81]  Dennis P. Wall,et al.  The Potential of Accelerating Early Detection of Autism through Content Analysis of YouTube Videos , 2014, PloS one.

[82]  Laura Pérez-Cano,et al.  Inherited and De Novo Genetic Risk for Autism Impacts Shared Networks , 2019, Cell.

[83]  Emad Barsoum,et al.  Training deep networks for facial expression recognition with crowd-sourced label distribution , 2016, ICMI.

[84]  Vanessa Moore,et al.  How Well Does Early Diagnosis of Autism Stand the Test of Time? , 2003, Autism : the international journal of research and practice.

[85]  Jean-Philippe Thiran,et al.  Analysis of Head-Mounted Wireless Camera Videos for Early Diagnosis of Autism , 2008, Computer Recognition Systems 2.

[86]  Kim M. Dalton,et al.  Gaze fixation and the neural circuitry of face processing in autism , 2005, Nature Neuroscience.

[87]  Peter Washington,et al.  Effect of Wearable Digital Intervention for Improving Socialization in Children With Autism Spectrum Disorder: A Randomized Clinical Trial , 2019, JAMA pediatrics.

[88]  A. Volein,et al.  Prediction of Autism at 3 Years from Behavioural and Developmental Measures in High-Risk Infants: A Longitudinal Cross-Domain Classifier Analysis , 2018, Journal of autism and developmental disorders.

[89]  Vibha Sazawal,et al.  Virtual Reality Support for Joint Attention Using the Floreo Joint Attention Module: Usability and Feasibility Pilot Study , 2019, JMIR pediatrics and parenting.

[90]  Duy Nguyen,et al.  AMP: A platform for managing and mining data in the treatment of Autism Spectrum Disorder , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[91]  Robert T. Schultz,et al.  Using natural conversations to classify autism with limited data: Age matters , 2019, Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology.

[92]  Peter Washington,et al.  Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism , 2018, npj Digital Medicine.

[93]  A. Klin,et al.  Infant viewing of social scenes is under genetic control and atypical in autism , 2017, Nature.

[94]  Guillermo Sapiro,et al.  Atypical postural control can be detected via computer vision analysis in toddlers with autism spectrum disorder , 2018, Scientific Reports.

[95]  Dennis P. Wall,et al.  Machine learning for early detection of autism (and other conditions) using a parental questionnaire and home video screening , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[96]  Janet B W Williams,et al.  Diagnostic and Statistical Manual of Mental Disorders , 2013 .

[97]  Sven Bölte,et al.  Joint attention in infancy and the emergence of autism , 2019, Biological Psychiatry.

[98]  Peter Washington,et al.  A Gamified Mobile System for Crowdsourcing Video for Autism Research , 2018, 2018 IEEE International Conference on Healthcare Informatics (ICHI).

[99]  Finale Doshi-Velez,et al.  Electronic Health Record Based Algorithm to Identify Patients with Autism Spectrum Disorder , 2016, PloS one.

[100]  Maarten De Vos,et al.  Look me in the eye: evaluating the accuracy of smartphone-based eye tracking for potential application in autism spectrum disorder research , 2019, Biomedical engineering online.

[101]  D. Wall,et al.  Crowdsourced validation of a machine-learning classification system for autism and ADHD , 2017, Translational Psychiatry.

[102]  P. Robinson,et al.  The emotional hearing aid: an assistive tool for children with Asperger syndrome , 2005, Universal Access in the Information Society.

[103]  Matthew S. Goodwin,et al.  Use of machine learning to improve autism screening and diagnostic instruments: effectiveness, efficiency, and multi-instrument fusion. , 2016, Journal of child psychology and psychiatry, and allied disciplines.

[104]  Scott P. Johnson,et al.  18-month predictors of later outcomes in younger siblings of children with autism spectrum disorder: a baby siblings research consortium study. , 2014, Journal of the American Academy of Child and Adolescent Psychiatry.

[105]  D. Wall,et al.  Use of Artificial Intelligence to Shorten the Behavioral Diagnosis of Autism , 2012, PloS one.

[106]  Emily M. Lund Social Communication Questionnaire , 2014 .

[107]  Andrew Pickles,et al.  Measuring Changes in Social Communication Behaviors: Preliminary Development of the Brief Observation of Social Communication Change (BOSCC) , 2016, Journal of autism and developmental disorders.

[108]  Declan G. M. Murphy,et al.  Identification and validation of biomarkers for autism spectrum disorders , 2015, Nature Reviews Drug Discovery.

[109]  Peter Washington,et al.  SuperpowerGlass , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[110]  Georgina Peacock,et al.  Whittling Down the Wait Time: Exploring Models to Minimize the Delay from Initial Concern to Diagnosis and Treatment of Autism Spectrum Disorder. , 2016, Pediatric clinics of North America.

[111]  Anibal Gutierrez,et al.  Objective measurement of head movement differences in children with and without autism spectrum disorder , 2018, Molecular Autism.

[112]  Sebastien Levy,et al.  Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism , 2017, Molecular Autism.

[113]  A. Robert Calderbank,et al.  A scalable app for measuring autism risk behaviors in young children: A technical validity and feasibility study , 2015, EAI Endorsed Trans. Scalable Inf. Syst..

[114]  D. Wall,et al.  Clinical Evaluation of a Novel and Mobile Autism Risk Assessment , 2016, Journal of Autism and Developmental Disorders.

[115]  Lili He,et al.  Enhancing Diagnosis of Autism With Optimized Machine Learning Models and Personal Characteristic Data , 2019, Front. Comput. Neurosci..

[116]  Alexa T. McCray,et al.  Modeling the Autism Spectrum Disorder Phenotype , 2013, Neuroinformatics.

[117]  Zachary Warren,et al.  Screening in toddlers and preschoolers at risk for autism spectrum disorder: Evaluating a novel mobile‐health screening tool , 2018, Autism research : official journal of the International Society for Autism Research.

[118]  Areej Al-Wabil,et al.  Intelligent Gaze-Based Screening System for Autism , 2019, 2019 2nd International Conference on Computer Applications & Information Security (ICCAIS).

[119]  Pamela Ventola,et al.  A Facial Affect Analysis System for Autism Spectrum Disorder , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[120]  C. Corsello,et al.  Early Intervention in Autism , 2005 .

[121]  F. Amthor,et al.  Avatar Assistant: Improving Social Skills in Students with an ASD Through a Computer-Based Intervention , 2011, Journal of autism and developmental disorders.