Accuracy and Reliability of At-Home Quantification of Motor Impairments Using a Computer-Based Pointing Task with Children with Ataxia-Telangiectasia

Methods for obtaining accurate quantitative assessments of motor impairments are essential in accessibility research, design of adaptive ability-based assistive technologies, as well as in clinical care and medical research. Currently, such assessments are typically performed in controlled laboratory or clinical settings under professional supervision. Emerging approaches for collecting data in unsupervised settings have been shown to produce valid data when aggregated over large populations, but it is not yet established whether in unsupervised settings measures of research or clinical significance can be collected accurately and reliably for individuals. We conducted a study with 13 children with ataxia-telangiectasia and 9 healthy children to analyze the validity, test-retest reliability, and acceptability of at-home use of a recent active digital phenotyping system, called Hevelius. Hevelius produces 32 measures derived from the movement trajectories of the mouse cursor and then generates a quantitative estimate of motor impairment in the dominant arm using the dominant arm component of the Brief Ataxia Rating Scale (BARS). The severity score estimates generated by Hevelius from single at-home sessions deviated from clinician-assigned BARS scores more than the severity score estimates generated from single sessions conducted under researcher supervision. However, taking a median of as few as 2 consecutive sessions produced severity score estimates that were as accurate or better than the estimates produced from single supervised sessions. Further, aggregating as few as 2 consecutive sessions resulted in good test-retest reliability (ICC = 0.81 for A-T participants). This work demonstrated the feasibility of performing accurate and reliable quantitative assessments of individual motor impairments in the dominant arm through tasks performed at home without supervision by the researchers. Further work is needed, however, to assess how broadly these results generalize.

[1]  Anoopum S. Gupta,et al.  Real-life Wrist Movement Patterns Capture Motor Impairment in Individuals with Ataxia-Telangiectasia , 2022, The Cerebellum.

[2]  D. Allen,et al.  Remote Assessments of Hand Function in Neurological Disorders: Systematic Review , 2022, JMIR rehabilitation and assistive technologies.

[3]  C. M. Verrelli,et al.  Gaming Technology for Pediatric Neurorehabilitation: A Systematic Review , 2022, Frontiers in Pediatrics.

[4]  Anoopum S. Gupta.  Digital Phenotyping in Clinical Neurology , 2022, Seminars in Neurology.

[5]  S. Schaefer,et al.  Remote, Unsupervised Functional Motor Task Evaluation in Older Adults across the United States Using the MindCrowd Electronic Cohort , 2021, Developmental neuropsychology.

[6]  Anoopum S. Gupta,et al.  Free-Living Motor Activity Monitoring in Ataxia-Telangiectasia , 2021, The Cerebellum.

[7]  S. Schaefer,et al.  Remote, unsupervised functional motor task evaluation in older adults across the United States using the MindCrowd electronic cohort , 2021, medRxiv.

[8]  Leah Findlater,et al.  Input Accessibility: A Large Dataset and Summary Analysis of Age, Motor Ability and Input Performance , 2020, ASSETS.

[9]  M. de Vos,et al.  Smartphone-based remote assessment of upper extremity function for multiple sclerosis using the Draw a Shape Test , 2020, Physiological measurement.

[10]  Jeffrey M. Hausdorff,et al.  Long-term unsupervised mobility assessment in movement disorders , 2020, The Lancet Neurology.

[11]  Bernd Huber,et al.  Conducting online virtual environment experiments with uncompensated, unsupervised samples , 2020, PloS one.

[12]  Katharina Reinecke,et al.  Controlling for Participants’ Viewing Distance in Large-Scale, Psychophysical Online Experiments Using a Virtual Chinrest , 2020, Scientific Reports.

[13]  M. Mildner,et al.  Re-epithelialization and immune cell behaviour in an ex vivo human skin model , 2020, Scientific Reports.

[14]  Katharina Reinecke,et al.  Computer mouse use captures ataxia and parkinsonism, enabling accurate measurement and detection , 2019, Movement disorders : official journal of the Movement Disorder Society.

[15]  Krzysztof Z. Gajos,et al.  "I think we know more than our doctors" , 2019, Proc. ACM Hum. Comput. Interact..

[16]  Michele Matarazzo,et al.  Remote Monitoring of Treatment Response in Parkinson's Disease: The Habit of Typing on a Computer , 2019, Movement disorders : official journal of the Movement Disorder Society.

[17]  Katharina Reinecke,et al.  Volunteer-Based Online Studies With Older Adults and People with Disabilities , 2018, ASSETS.

[18]  Krzysztof Z. Gajos,et al.  Ability-based design , 2018, Commun. ACM.

[19]  Frédo Durand,et al.  A Video-Based Method for Automatically Rating Ataxia , 2017, MLHC.

[20]  Katharina Reinecke,et al.  The Effect of Performance Feedback on Social Media Sharing at Volunteer-Based Online Experiment Platforms , 2017, CHI.

[21]  Jon Froehlich,et al.  Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data , 2017, CHI.

[22]  Kyle Montague,et al.  Investigating Laboratory and Everyday Typing Performance of Blind Users , 2017, ACM Trans. Access. Comput..

[23]  T. Crawford,et al.  Ataxia telangiectasia: a review , 2016, Orphanet Journal of Rare Diseases.

[24]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[25]  Krzysztof Z. Gajos,et al.  Curiosity Killed the Cat, but Makes Crowdwork Better , 2016, CHI.

[26]  S. Rauch,et al.  Harnessing Smartphone-Based Digital Phenotyping to Enhance Behavioral and Mental Health , 2016, Neuropsychopharmacology.

[27]  B. Ben-Zeev,et al.  Ataxia telangiectasia. , 2015, Handbook of clinical neurology.

[28]  Max A. Little,et al.  Detecting and monitoring the symptoms of Parkinson's disease using smartphones: A pilot study. , 2015, Parkinsonism & related disorders.

[29]  Brian W. Powers,et al.  The digital phenotype , 2015, Nature Biotechnology.

[30]  M. Horne,et al.  An Objective Fluctuation Score for Parkinson's Disease , 2015, PloS one.

[31]  Katharina Reinecke,et al.  LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples , 2015, CSCW.

[32]  R. Adelman,et al.  Caregiver burden: a clinical review. , 2014, JAMA.

[33]  Scott E. Hudson,et al.  Distinguishing Users By Pointing Performance in Laboratory and Real-World Tasks , 2013, TACC.

[34]  D. Zee,et al.  Disorders of Upper Limb Movements in Ataxia-Telangiectasia , 2013, PloS one.

[35]  Randa L. Shehab,et al.  Effects of age and psychomotor ability on kinematics of mouse-mediated aiming movement , 2013, Ergonomics.

[36]  Katharina Reinecke,et al.  Crowdsourcing performance evaluations of user interfaces , 2013, CHI.

[37]  Etienne Burdet,et al.  A Robust and Sensitive Metric for Quantifying Movement Smoothness , 2012, IEEE Transactions on Biomedical Engineering.

[38]  K. Nakayama,et al.  Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments , 2012, Psychonomic Bulletin & Review.

[39]  Jacob O. Wobbrock,et al.  Taming wild behavior: the input observer for obtaining text entry and mouse pointing measures from everyday computer use , 2012, CHI.

[40]  Katharina Reinecke,et al.  Accurate measurements of pointing performance from in situ observations , 2012, CHI.

[41]  Krzysztof Z. Gajos,et al.  Personalized dynamic accessibility , 2012, INTR.

[42]  Krzysztof Z. Gajos,et al.  Automatically generating personalized user interfaces with Supple , 2010, Artif. Intell..

[43]  T. Groth,et al.  A new computer method for assessing drawing impairment in Parkinson's disease , 2010, Journal of Neuroscience Methods.

[44]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[45]  Dagmar Sternad,et al.  Sensitivity of Smoothness Measures to Movement Duration, Amplitude, and Arrests , 2009, Journal of motor behavior.

[46]  Jeremy D. Schmahmann,et al.  Development of a brief ataxia rating scale (BARS) based on a modified form of the ICARS , 2009, Movement disorders : official journal of the Movement Disorder Society.

[47]  Scott E. Hudson,et al.  Understanding pointing problems in real world computing environments , 2008, Assets '08.

[48]  Krzysztof Z. Gajos,et al.  Goal Crossing with Mice and Trackballs for People with Motor Impairments: Performance, Submovements, and Design Directions , 2008, TACC.

[49]  Scott E. Hudson,et al.  Automatically detecting pointing performance , 2008, IUI '08.

[50]  Afke Donker,et al.  Aiming and clicking in young children's use of the computer mouse , 2007, Comput. Hum. Behav..

[51]  Simeon Keates,et al.  Effect of age and Parkinson's disease on cursor positioning using a mouse , 2005, Assets '05.

[52]  T. Crawford,et al.  Survival probability in ataxia telangiectasia , 2005, Archives of Disease in Childhood.

[53]  Allison Druin,et al.  Differences in pointing task performance between preschool children and adults using mice , 2004, TCHI.

[54]  Eric R. Ziegel,et al.  Probability and Statistics for Engineering and the Sciences , 2004, Technometrics.

[55]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[56]  S. Gosling,et al.  Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. , 2004, The American psychologist.

[57]  Peter Robinson,et al.  Cursor measures for motion-impaired computer users , 2002, ASSETS.

[58]  I. Scott MacKenzie,et al.  Accuracy measures for evaluating computer pointing devices , 2001, CHI.

[59]  Elena Rocco,et al.  Trust breaks down in electronic contexts but can be repaired by some initial face-to-face contact , 1998, CHI.

[60]  N Walker,et al.  Spatial and Temporal Characteristics of Rapid Cursor-Positioning Movements with Electromechanical Mice in Human-Computer Interaction , 1993, Human factors.

[61]  R. Sakia.  The Box-Cox transformation technique: a review , 1992 .

[62]  J. Cooke,et al.  Kinematics of arm movements in elderly humans , 1989, Neurobiology of Aging.

[63]  Daniel W. Repperger,et al.  Fitts' law and the microstructure of rapid discrete movements. , 1980 .

[64]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[65]  P. Kempster,et al.  Automated assessment of bradykinesia and dyskinesia in Parkinson's disease. , 2012, Journal of Parkinson's disease.

[66]  Wolfgang Köpcke,et al.  Clinical trials and rare diseases. , 2010, Advances in experimental medicine and biology.

[67]  George E Stelmach,et al.  Age-related kinematic differences as influenced by task difficulty, target size, and movement amplitude. , 2002, The journals of gerontology. Series B, Psychological sciences and social sciences.

[68]  A. D. Fisk,et al.  Age-related differences in movement control: adjusting submovement structure to optimize performance. , 1997, The journals of gerontology. Series B, Psychological sciences and social sciences.

[69]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[70]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.