Evaluation, Acceptance, and Qualification of Digital Measures: From Proof of Concept to Endpoint

To support the successful adoption of digital measures into internal decision making and evidence generation for medical product development, we present a unified lexicon to aid communication throughout this process, and highlight key concepts including the critical role of participant engagement in development of digital measures. We detail the steps of bringing a successful proof of concept to scale, focusing on key decisions in the development of a new digital measure: asking the right question, optimized approaches to evaluating new measures, and whether and how to pursue qualification or acceptance. Building on the V3 framework for establishing verification and analytical and clinical validation, we discuss strategic and practical considerations for collecting this evidence, illustrated with concrete examples of trailblazing digital measures in the field.

[1]  Simon C. Mathews,et al.  A digital health industry cohort across the health continuum , 2020, npj Digital Medicine.

[2]  Joseph P Menetski,et al.  Remote digital monitoring in clinical trials in the time of COVID-19 , 2020, Nature Reviews Drug Discovery.

[3]  J. Goldsack,et al.  Defining and Developing the Workforce Needed for Success in the Digital Era of Medicine , 2020, Digital Biomarkers.

[4]  C. Berridge,et al.  Sensor-Based Passive Remote Monitoring and Discordant Values: Qualitative Study of the Experiences of Low-Income Immigrant Elders in the United States , 2018, JMIR mHealth and uHealth.

[5]  B. Caulfield,et al.  Patient-Generated Health Data , 2018 .

[6]  Ariel V. Dowling,et al.  Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs) , 2020, npj Digital Medicine.

[7]  A. Stern,et al.  Quantifying the use of connected digital products in clinical research , 2020, npj Digital Medicine.

[8]  Robert A Koeppe,et al.  Thalamic cholinergic innervation and postural sensory integration function in Parkinson's disease. , 2013, Brain : a journal of neurology.

[9]  Adriana M. Seelye,et al.  Computer mouse movement patterns: A potential marker of mild cognitive impairment , 2015, Alzheimer's & dementia.

[10]  Ieuan Clay,et al.  Impact of Digital Technologies on Novel Endpoint Capture in Clinical Trials , 2017, Clinical pharmacology and therapeutics.

[11]  Hongyu Luo,et al.  Assessment of Fatigue Using Wearable Sensors: A Pilot Study , 2020, Digital Biomarkers.

[12]  Ieuan Clay,et al.  The Path Forward for Digital Measures: Suppressing the Desire to Compare Apples and Pineapples , 2020, Digital Biomarkers.

[13]  Dimitrios Athanasiou,et al.  European regulators’ views on a wearable-derived performance measurement of ambulation for Duchenne muscular dystrophy regulatory trials , 2019, Neuromuscular Disorders.

[14]  R. Mehanna.  Gait speed in Parkinson disease correlates with cholinergic degeneration , 2014, Neurology.

[15]  Jennifer C. Goldsack,et al.  Digital Medicine: A Primer on Measurement , 2019, Digital Biomarkers.

[16]  L. Ferrucci,et al.  A short physical performance battery assessing lower extremity function: association with self-reported disability and prediction of mortality and nursing home admission. , 1994, Journal of gerontology.

[17]  S. Friend,et al.  The mPower study, Parkinson disease mobile data collected using ResearchKit , 2016, Scientific Data.

[18]  Tih-Shih Lee,et al.  Early Detection of Mild Cognitive Impairment With In-Home Sensors to Monitor Behavior Patterns in Community-Dwelling Senior Citizens in Singapore: Cross-Sectional Feasibility Study , 2019, Journal of medical Internet research.

[19]  Julia M. Leach,et al.  Day-to-Day Variability of Postural Sway and Its Association With Cognitive Function in Older Adults: A Pilot Study , 2018, Front. Aging Neurosci..

[20]  Philip Beineke,et al.  Developing Smartphone-Based Objective Assessments of Physical Function in Rheumatoid Arthritis Patients: The PARADE Study , 2020, Digital Biomarkers.

[21]  Richard W. Bohannon,et al.  Minimal clinically important difference for change in 6‐minute walk test distance of adults with pathology: a systematic review , 2017, Journal of evaluation in clinical practice.

[22]  Charlie Quinn,et al.  Methodology for Establishing a Community-Wide Life Laboratory for Capturing Unobtrusive and Continuous Remote Activity and Health Data. , 2018, Journal of visualized experiments : JoVE.

[23]  S. Tokuno,et al.  A validation study of a consumer wearable sleep tracker compared to a portable EEG system in naturalistic conditions. , 2019, Journal of psychosomatic research.

[24]  Philip D. Harvey,et al.  GPS mobility as a digital biomarker of negative symptoms in schizophrenia: a case control study , 2019, npj Digital Medicine.

[25]  Vera Maljkovic,et al.  Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams , 2019, KDD.

[26]  P. Bower,et al.  Trials need participants but not their feedback? A scoping review of published papers on the measurement of participant experience of taking part in clinical trials , 2019, Trials.

[27]  X. Montalban,et al.  Adherence and Satisfaction of Smartphone- and Smartwatch-Based Remote Active Testing and Passive Monitoring in People With Multiple Sclerosis: Nonrandomized Interventional Feasibility Study , 2019, Journal of medical Internet research.

[28]  P. Dagum.  Digital biomarkers of cognitive function , 2018, npj Digital Medicine.

[29]  Maarten De Vos,et al.  Look me in the eye: evaluating the accuracy of smartphone-based eye tracking for potential application in autism spectrum disorder research , 2019, Biomedical engineering online.

[30]  Christine Manta,et al.  Digital Measures That Matter to Patients: A Framework to Guide the Selection and Development of Digital Measures of Health , 2020, Digital Biomarkers.

[31]  Amir Muaremi,et al.  Continuous Digital Monitoring of Walking Speed in Frail Elderly Patients: Noninterventional Validation Study and Longitudinal Clinical Trial , 2019, JMIR mHealth and uHealth.

[32]  Stephanie L. Shimada,et al.  Psychometric Properties of Patient-Facing eHealth Evaluation Measures: Systematic Review and Analysis , 2017, Journal of medical Internet research.

[33]  Ieuan Clay,et al.  Validity of accelerometry in step detection and gait speed measurement in orthogeriatric patients , 2019, PloS one.

[34]  Francesca Cerreta,et al.  Digital technologies for medicines: shaping a framework for success , 2020, Nature reviews. Drug discovery.

[35]  Daniel Austin,et al.  Unobtrusive measurement of daily computer use to detect mild cognitive impairment , 2014, Alzheimer's & Dementia.

[36]  Suchi Saria,et al.  Using Smartphones and Machine Learning to Quantify Parkinson Disease Severity: The Mobile Parkinson Disease Score , 2018, JAMA neurology.

[37]  K. Flanigan,et al.  How a patient advocacy group developed the first proposed draft guidance document for industry for submission to the U.S. Food and Drug Administration , 2015, Orphanet Journal of Rare Diseases.

[38]  Akane Sano,et al.  Stress Recognition Using Wearable Sensors and Mobile Phones , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[39]  Beatrix Vereijken,et al.  A Roadmap to Inform Development, Validation and Approval of Digital Mobility Outcomes: The Mobilise-D Approach , 2020, Digital Biomarkers.

[40]  Sydney Katz.  Assessing Self‐maintenance: Activities of Daily Living, Mobility, and Instrumental Activities of Daily Living , 1983, Journal of the American Geriatrics Society.

[41]  Qi Liu,et al.  Remote Digital Monitoring for Medical Product Development , 2020, Clinical and Translational Science.

[42]  Richard Dobson,et al.  RADAR-Base: Open Source Mobile Health Platform for Collecting, Monitoring, and Analyzing Data Using Sensors, Wearables, and Mobile Devices , 2019, JMIR mHealth and uHealth.

[43]  Arno Klein,et al.  Personalized Hypothesis Tests for Detecting Medication Response in Parkinson Disease Patients Using iPhone Sensor Data , 2016, PSB.

[44]  I. Clay,et al.  Predicting Subjective Recovery from Lower Limb Surgery Using Consumer Wearables , 2020, Digital Biomarkers.

[45]  V. Ricotti,et al.  Respiratory and upper limb function as outcome measures in ambulant and non-ambulant subjects with Duchenne muscular dystrophy: A prospective multicentre study , 2019, Neuromuscular Disorders.

[46]  T. Hayes,et al.  One walk a year to 1000 within a year: continuous in-home unobtrusive gait assessment of older adults. , 2012, Gait & posture.

[47]  Elizabeth Molsen,et al.  Clinical Outcome Assessments: Conceptual Foundation-Report of the ISPOR Clinical Outcomes Assessment - Emerging Good Practices for Outcomes Research Task Force. , 2015, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[48]  Martin Jaggi,et al.  Prediction of Patient-Reported Physical Activity Scores from Wearable Accelerometer Data: A Feasibility Study , 2018, Converging Clinical and Engineering Research on Neurorehabilitation III.

[49]  J. Manson,et al.  36‐Item Short Form Survey (SF‐36) Versus Gait Speed As Predictor of Preclinical Mobility Disability in Older Women: The Women's Health Initiative , 2018, Journal of the American Geriatrics Society.

[50]  S. Michie,et al.  Assessing the Psychometric Properties of the Digital Behavior Change Intervention Engagement Scale in Users of an App for Reducing Alcohol Consumption: Evaluation Study , 2019, Journal of medical Internet research.

[51]  Luca Foschini,et al.  Using Wearable Devices and Smartphones to Track Physical Activity: Initial Activation, Sustained Use, and Step Counts Across Sociodemographic Characteristics in a National Sample , 2017, Annals of Internal Medicine.

[52]  Sarah Gothard,et al.  The Collaborative Aging Research Using Technology Initiative: An Open, Sharable, Technology-Agnostic Platform for the Research Community , 2020, Digital Biomarkers.

[53]  Antoine Piau,et al.  Current State of Digital Biomarker Technologies for Real-Life, Home-Based Monitoring of Cognitive Function for Mild Cognitive Impairment to Mild Alzheimer Disease and Implications for Clinical Care: Systematic Review , 2019, Journal of medical Internet research.

[54]  N. Jacobson,et al.  Digital biomarkers of mood disorders and symptom change , 2019, npj Digital Medicine.

[55]  L. Foschini,et al.  Continuous Digital Assessment for Weight Loss Surgery Patients , 2020, Digital Biomarkers.

[56]  C. Blake,et al.  The reliability of the quantitative timed up and go test (QTUG) measured over five consecutive days under single and dual-task conditions in community dwelling older adults. , 2016, Gait & posture.

[57]  Elena Smets,et al.  Large-scale wearable data reveal digital phenotypes for daily-life stress detection , 2018, npj Digital Medicine.

[58]  Chi Heem Wong,et al.  Estimation of clinical trial success rates and related parameters , 2018, Biostatistics.

[59]  Dina Katabi,et al.  Passive Monitoring at Home: A Pilot Study in Parkinson Disease , 2019, Digital Biomarkers.

[60]  Beau Woods,et al.  Modernizing and designing evaluation frameworks for connected sensor technologies in medicine , 2020, npj Digital Medicine.

[61]  T. Fleming,et al.  Biomarkers and surrogate endpoints in clinical trials , 2012, Statistics in medicine.