Automatic Detection of Amyotrophic Lateral Sclerosis (ALS) from Video-Based Analysis of Facial Movements: Speech and Non-Speech Tasks

The analysis of facial movements in patients with amyotrophic lateral sclerosis (ALS) can provide important information about early diagnosis and tracking disease progression. However, the use of expensive motion tracking systems has limited the clinical utility of the assessment. In this study, we propose a marker-less video-based approach to discriminate patients with ALS from neurotypical subjects. Facial movements were recorded using a depth sensor (Intel® RealSense" SR300) during speech and nonspeech tasks. A small set of kinematic features of lips was extracted in order to mirror the perceptual evaluation performed by clinicians, considering the following aspects: (1) range of motion, (2) speed of motion, (3) symmetry, and (4) shape. Our results demonstrate that it is possible to distinguish patients with ALS from neurotypical subjects with high overall accuracy (up to 88.9%) during repetitions of sentences, syllables, and labial non-speech movements (e.g., lip spreading). This paper provides strong rationale for the development of automated systems to detect neurological diseases from facial movements. This work has a high social impact, as it opens new possibilities to develop intelligent systems to support clinicians in their diagnosis, introducing novel standards for assessing the oro-facial impairment in ALS, and tracking disease progression remotely from home.

[1]  Horst-Michael Groß,et al.  Automated Classification of Therapeutic Face Exercises using the Kinect , 2013, VISAPP.

[2]  G. Zaccara,et al.  Markerless Analysis of Articulatory Movements in Patients With Parkinson's Disease. , 2016, Journal of voice : official journal of the Voice Foundation.

[3]  Max A. Little,et al.  Novel Speech Signal Processing Algorithms for High-Accuracy Classification of Parkinson's Disease , 2012, IEEE Transactions on Biomedical Engineering.

[4]  Matthew C Kiernan,et al.  Palliative care in amyotrophic lateral sclerosis , 2015, The Lancet Neurology.

[5]  Yana Yunusova,et al.  Classification of Bulbar ALS from Kinematic Features of the Jaw and Lips: Towards Computer-Mediated Assessment , 2017, INTERSPEECH.

[6]  Tim K Marks,et al.  Robust Face Alignment Using a Mixture of Invariant Experts , 2016, ECCV.

[7]  Daphna Weinshall,et al.  Quantifying Hypomimia in Parkinson Patients Using a Depth Camera , 2015, MindCare.

[8]  Bogdan J. Matuszewski,et al.  Facial asymmetry analysis based on 3-D dynamic scans , 2012, 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[9]  Ales Procházka,et al.  Face movement analysis with MS Kinect , 2016, 2016 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM).

[10]  P. Hogan,et al.  Cost of illness for neuromuscular diseases in the United States , 2014, Muscle & nerve.

[11]  Panying Rong,et al.  Predicting Early Bulbar Decline in Amyotrophic Lateral Sclerosis: A Speech Subsystem Approach , 2015, Behavioural neurology.

[12]  Cheng Li,et al.  Face alignment by coarse-to-fine shape searching , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[14]  Panying Rong,et al.  Speech Movement Measures as Markers of Bulbar Disease in Amyotrophic Lateral Sclerosis. , 2016, Journal of speech, language, and hearing research : JSLHR.

[15]  H. Mitsumoto,et al.  Improving the quality of life for people with ALS: The challenge ahead , 2000, Amyotrophic lateral sclerosis and other motor neuron disorders : official publication of the World Federation of Neurology, Research Group on Motor Neuron Diseases.

[16]  Joseph R. Duffy,et al.  Motor Speech Disorders: Clues to Neurologic Diagnosis , 2000 .

[17]  S. Langmore,et al.  Physiologic deficits in the orofacial system underlying dysarthria in amyotrophic lateral sclerosis. , 1994, Journal of speech and hearing research.

[18]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[19]  Horst-Michael Groß,et al.  A Feedback Estimation Approach for Therapeutic Facial Training , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[20]  J. Cedarbaum,et al.  The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function , 1999, Journal of the Neurological Sciences.

[21]  Jordan R. Green,et al.  Speaking rate effects on articulatory pattern consistency in talkers with mild ALS , 2014, Clinical linguistics & phonetics.

[22]  B. Crum,et al.  The diagnostic pathway and prognosis in bulbar-onset amyotrophic lateral sclerosis , 2010 .

[23]  Jun Wang,et al.  Towards Automatic Detection of Amyotrophic Lateral Sclerosis from Speech Acoustic and Articulatory Samples , 2016, INTERSPEECH.

[24]  J. Caroscio,et al.  Amyotrophic lateral sclerosis. Its natural history. , 1987, Neurologic clinics.

[25]  Claudia Manfredi,et al.  Analysis of facial expressions in parkinson's disease through video-based automatic methods , 2017, Journal of Neuroscience Methods.

[26]  Dongmei Jiang,et al.  Objectifying Facial Expressivity Assessment of Parkinson's Patients: Preliminary Study , 2014, Comput. Math. Methods Medicine.

[27]  Bernhard Neundörfer,et al.  Subjective experience and coping in ALS , 2002, Amyotrophic lateral sclerosis and other motor neuron disorders : official publication of the World Federation of Neurology, Research Group on Motor Neuron Diseases.

[28]  Gary L. Pattee,et al.  Bulbar and speech motor assessment in ALS: Challenges and future directions , 2013, Amyotrophic lateral sclerosis & frontotemporal degeneration.

[29]  Tomas E. Ward,et al.  A preliminary investigation into the use of machine vision techniques for automating facial paralysis rehabilitation therapy , 2010 .

[30]  J. Cummings,et al.  The Montreal Cognitive Assessment, MoCA: A Brief Screening Tool For Mild Cognitive Impairment , 2005, Journal of the American Geriatrics Society.

[31]  Laura J. Ball,et al.  Kinematics of disease progression in bulbar ALS. , 2010, Journal of communication disorders.

[32]  M. Lindstrom,et al.  Articulatory movements during vowels in speakers with dysarthria and healthy controls. , 2008, Journal of speech, language, and hearing research : JSLHR.

[33]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Panying Rong,et al.  Predicting Speech Intelligibility Decline in Amyotrophic Lateral Sclerosis Based on the Deterioration of Individual Speech Subsystems , 2016, PloS one.

[35]  Yana Yunusova,et al.  Classifications of vocalic segments from articulatory kinematics: healthy controls and speakers with dysarthria. , 2011, Journal of speech, language, and hearing research : JSLHR.

[36]  Yana Yunusova,et al.  The diagnostic utility of patient-report and speech-language pathologists’ ratings for detecting the early onset of bulbar symptoms due to ALS , 2017, Amyotrophic lateral sclerosis & frontotemporal degeneration.

[37]  Yana Yunusova,et al.  Kinematic Features of Jaw and Lips Distinguish Symptomatic From Presymptomatic Stages of Bulbar Decline in Amyotrophic Lateral Sclerosis. , 2018, Journal of speech, language, and hearing research : JSLHR.

[38]  C. Gaelyn Garrett,et al.  Otolaryngologic presentations of amyotrophic lateral sclerosis , 2005, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[39]  M. Swash,et al.  El Escorial revisited: Revised criteria for the diagnosis of amyotrophic lateral sclerosis , 2000, Amyotrophic lateral sclerosis and other motor neuron disorders : official publication of the World Federation of Neurology, Research Group on Motor Neuron Diseases.

[40]  Elizabeth K. Hanson,et al.  Dysarthria in Amyotrophic Lateral Sclerosis: A Systematic Review of Characteristics, Speech Treatment, and Augmentative and Alternative Communication Options , 2011 .

[41]  Yana Yunusova,et al.  A Protocol for Comprehensive Assessment of Bulbar Dysfunction in Amyotrophic Lateral Sclerosis (ALS) , 2011, Journal of visualized experiments : JoVE.

[42]  Jordan R. Green,et al.  A novel fixed-target task to determine articulatory speed constraints in persons with amyotrophic lateral sclerosis. , 2012, Journal of communication disorders.