Data labelling in the wild: annotating free-living activities and Parkinson's disease symptoms

This paper looks to explore the challenges faced when producing a set of annotations from videos produced by a pilot study evaluating 24 participants (12 with Parkinson's disease, each accompanied by a healthy volunteer control participant) who are free-living in a house embedded with a platform of sensors. We discuss the outcome measures chosen to annotate from the videos and the controlled vocabularies formulated for this task, the tools and processes, how we intend to achieve standardisation and normalisation of the annotations, and how to improve quality and re-usability of the annotation dataset.

[1]  J. Jankovic,et al.  Movement Disorder Society‐sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS‐UPDRS): Scale presentation and clinimetric testing results , 2008, Movement disorders : official journal of the Movement Disorder Society.

[2]  Andrew M. Dai,et al.  Wearable sensors for Parkinson’s disease: which data are worth collecting for training symptom detection models , 2018, npj Digital Medicine.

[3]  T. Vos,et al.  Global, regional, and national incidence and prevalence, and years lived with disability for 328 diseases and injuries in 195 countries, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016 , 2017 .

[4]  Verónica Robles-García,et al.  Spatiotemporal Gait Patterns During Overt and Covert Evaluation in Patients With Parkinson´s Disease and Healthy Subjects: Is There a Hawthorne Effect? , 2015, Journal of applied biomechanics.

[5]  Chengjie Xiong,et al.  Unified Parkinson's Disease Rating Scale-Motor Exam: inter-rater reliability of advanced practice nurse and neurologist assessments. , 2010, Journal of advanced nursing.

[6]  O. Lindvall,et al.  Use and interpretation of on/off diaries in Parkinson’s disease , 2004, Journal of Neurology, Neurosurgery & Psychiatry.

[7]  Niall Twomey,et al.  A Comprehensive Study of Activity Recognition Using Accelerometers , 2018, Informatics.

[8]  L. Rochester,et al.  Systematic Review Looking at the Use of Technology to Measure Free-Living Symptom and Activity Outcomes in Parkinson’s Disease in the Home or a Home-like Environment , 2020, Journal of Parkinson's disease.

[9]  Gbd Disease and Injury Incidence and Prevalence Collab.  Global, regional, and national incidence, prevalence, and years lived with disability for 328 diseases and injuries for 195 countries, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016 , 2017 .

[10]  Milos Hauskrecht,et al.  Learning classification models from multiple experts , 2013, J. Biomed. Informatics.

[11]  Niall Twomey,et al.  Talk, Text, Tag? Understanding Self-Annotation of Smart Home Data from a User’s Perspective , 2018, Sensors.

[12]  Raúl Santos-Rodríguez,et al.  Understanding the quality of calibrations for indoor localisation , 2018, 2018 IEEE 4th World Forum on Internet of Things (WF-IoT).

[13]  Antonis Vafeas,et al.  Towards a Methodology for Acceptance Testing and Validation of Monitoring Bodyworn Devices , 2020, 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[14]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[15]  Geoffrey E. Hinton,et al.  Who Said What: Modeling Individual Labelers Improves Classification , 2017, AAAI.

[16]  W. Poewe,et al.  Trial designs used to study neuroprotective therapy in Parkinson's disease , 2013, Movement disorders : official journal of the Movement Disorder Society.

[17]  Hennie Brugman,et al.  Annotating Multi-media/Multi-modal Resources with ELAN , 2004, LREC.

[18]  Kamlesh Khunti,et al.  Intensity Thresholds on Raw Acceleration Data: Euclidean Norm Minus One (ENMO) and Mean Amplitude Deviation (MAD) Approaches , 2016, PloS one.

[19]  Niall Twomey,et al.  SPHERE: A sensor platform for healthcare in a residential environment , 2017 .

[20]  P. Bossuyt,et al.  Evaluation of diagnostic tests when there is no gold standard. A review of methods. , 2007, Health technology assessment.

[21]  Peter A. Flach,et al.  Activities of Daily Living Ontology for Ubiquitous Systems: Development and Evaluation , 2018, Sensors.

[22]  Gerardo Hermosillo,et al.  Supervised learning from multiple experts: whom to trust when everyone lies a bit , 2009, ICML '09.

[23]  Ian Craddock,et al.  Towards estimation of cooking complexity: free-text annotations in the kitchen environment , 2019, iWOAR.

[24]  Ian Craddock,et al.  A dataset for room level indoor localization using a smart home in a box , 2019, Data in brief.

[25]  Niall Twomey,et al.  Bridging e-Health and the Internet of Things: The SPHERE Project , 2015, IEEE Intelligent Systems.