Reliable data collection in participatory trials to assess digital healthcare apps

The number of digital healthcare mobile apps on the market is increasing exponentially owing to the development of the mobile network and widespread usage of smartphones. However, only a few of these apps have undergone adequate validation. As with many mobile apps, healthcare apps are generally considered safe to use, making them easy for developers and end-users to exchange them in the marketplace. The existing platforms are not suitable to collect reliable data for evaluating the effectiveness of the apps. Moreover, these platforms only reflect the perspectives of developers and experts, not of end-users. For instance, data collection methods typical of clinical trials are not appropriate for participant-driven assessment of healthcare apps because of their complexity and high cost. Thus, we identified a need for a participant-driven data collection platform for end-users that is interpretable, systematic, and sustainable —as a first step to validate the effectiveness of the apps. To collect reliable data in the participatory trial format, we defined distinct stages for data preparation, storage, and sharing. Interpretable data preparation consists of a protocol database system and semantic feature retrieval method to create a protocol without professional knowledge. Collected data reliability weight calculation belongs to the systematic data storage stage. For sustainable data collection, we integrated the weight method and the future reward distribution function. We validated the methods through statistical tests conducted on 718 human participants. The validation results demonstrate that the methods have significant differences in the comparative experiment and prove that the choice of the right method is essential for reliable data collection. Furthermore, we created a web-based system for our pilot platform to collect reliable data in an integrated pipeline. We validate the platform features with existing clinical and pragmatic trial data collection platforms. In conclusion, we show that the method and platform support reliable data collection, forging a path to effectiveness validation of digital healthcare apps.

[1]  P. Costa,et al.  The revised NEO personality inventory (NEO-PI-R) , 2008 .

[2]  D. Alter,et al.  Uptake of an Incentive-Based mHealth App: Process Evaluation of the Carrot Rewards App , 2017, JMIR mHealth and uHealth.

[3]  R. McGrath,et al.  Evidence for response bias as a source of error variance in applied assessment. , 2010, Psychological bulletin.

[4]  Sean Khozin,et al.  Developing and adopting safe and effective digital biomarkers to improve patient outcomes , 2019, npj Digital Medicine.

[5]  J. Chung,et al.  Sustainable Growth and Token Economy Design: The Case of Steemit , 2018, Sustainability.

[6]  Kenneth A Getz,et al.  New Benchmarks Characterizing Growth in Protocol Design Complexity , 2018, Therapeutic innovation & regulatory science.

[7]  K. Volpp,et al.  A Pragmatic Trial of E‐Cigarettes, Incentives, and Drugs for Smoking Cessation , 2018, The New England journal of medicine.

[8]  J. Ezekowitz,et al.  Trends in the Explanatory or Pragmatic Nature of Cardiovascular Clinical Trials Over 2 Decades. , 2019, JAMA cardiology.

[9]  日経BP社,et al.  Amazon Web Services完全ソリューションガイド , 2016 .

[10]  Yuanyuan Zhang,et al.  The App Sampling Problem for App Store Mining , 2015, 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories.

[11]  D J Torgerson,et al.  Pragmatic trials: lab meets bedside , 2019, The British journal of dermatology.

[12]  Martin F Mendiola,et al.  Valuable Features in Mobile Health Apps for Patients and Consumers: Content Analysis of Apps and User Ratings , 2015, JMIR mHealth and uHealth.

[13]  S. Koike,et al.  Wearing blue light-blocking glasses in the evening advances circadian rhythms in the patients with delayed sleep phase disorder: An open-label trial , 2016, Chronobiology international.

[14]  Jason L. Huang,et al.  Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant Reactions , 2014, Journal of Business and Psychology.

[15]  L. Sweeney.  Simple Demographics Often Identify People Uniquely , 2000 .

[16]  Victor H. Vroom.  Some Personality Determinants of the Effects of Participation , 2019 .

[17]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[18]  Luohua Jiang,et al.  Long-Term Outcomes of a Web-Based Diabetes Prevention Program: 2-Year Results of a Single-Arm Longitudinal Study , 2015, Journal of medical Internet research.

[19]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[20]  Mircea Cinteza,et al.  A multifaceted intervention to improve treatment with oral anticoagulants in atrial fibrillation (IMPACT-AF): an international, cluster-randomised trial , 2017, The Lancet.

[21]  J. Greene,et al.  Assessing the Gold Standard--Lessons from the History of RCTs. , 2016, The New England journal of medicine.

[22]  C. Pannucci,et al.  Identifying and Avoiding Bias in Research , 2010, Plastic and reconstructive surgery.

[23]  David K. Lohrmann,et al.  Effects of preventive online mindfulness interventions on stress and mindfulness: A meta-analysis of randomized controlled trials , 2016, Preventive medicine reports.

[24]  E. Elenko,et al.  A regulatory framework emerges for digital medicine , 2015, Nature Biotechnology.

[25]  K. Plangger,et al.  Little rewards, big changes: Using exercise analytics to motivate sustainable changes in physical activity , 2019, Inf. Manag..

[26]  Robert H. Deng,et al.  CrowdBC: A Blockchain-Based Decentralized Framework for Crowdsourcing , 2019, IEEE Transactions on Parallel and Distributed Systems.

[27]  Bethany S. Jurs,et al.  How serious is the ‘carelessness’ problem on Mechanical Turk? , 2019, International Journal of Social Research Methodology.

[28]  G. Dawson,et al.  Brief Report: Pilot Study of a Novel Interactive Digital Treatment to Improve Cognitive Control in Children with Autism Spectrum Disorder and Co-occurring ADHD Symptoms , 2019, Journal of autism and developmental disorders.

[29]  Thomas Martin Deserno,et al.  DICOM for Clinical Research: PACS-Integrated Electronic Data Capture in Multi-Center Trials , 2015, Journal of Digital Imaging.

[30]  Philipp Neuhaus,et al.  ODM Data Analysis—A tool for the automatic validation, monitoring and generation of generic descriptive statistics of patient data , 2018, PloS one.

[31]  Alla Keselman,et al.  Registering a clinical trial in ClinicalTrials.gov. , 2007, Chest.

[32]  Matthew D. Galsky,et al.  Use of crowdsourcing for cancer clinical trial development. , 2014 .

[33]  Chris Callison-Burch,et al.  A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk , 2017, CHI.

[34]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[35]  Janice Branson,et al.  Protecting patient privacy when sharing patient-level data from clinical trials , 2016, BMC Medical Research Methodology.

[36]  K. McGraw,et al.  A common language effect size statistic. , 1992 .

[37]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[38]  D. Rennie,et al.  SPIRIT 2013 statement: defining standard protocol items for clinical trials. , 2013, Annals of internal medicine.

[39]  Seongkuk Park,et al.  An interactive retrieval system for clinical trial studies with context-dependent protocol elements , 2019, bioRxiv.

[40]  Nima Nikzad,et al.  Self-Monitoring Utilization Patterns Among Individuals in an Incentivized Program for Healthy Behaviors , 2016, Journal of medical Internet research.

[41]  R. W. Hansen,et al.  Journal of Health Economics , 2016 .

[42]  A. Šimundić.  Bias in research , 2013, Biochemia medica.

[43]  Daniel J. Veit,et al.  More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk , 2011, AMCIS.

[44]  Ross Tonkens,et al.  An overview of the drug development process. , 2005, Physician executive.

[45]  HP Selker,et al.  PIPELINEs: Creating Comparable Clinical Knowledge Efficiently by Linking Trial Platforms , 2016, Clinical pharmacology and therapeutics.

[46]  S. Friend,et al.  The mPower study, Parkinson disease mobile data collected using ResearchKit , 2016, Scientific Data.

[47]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .

[48]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[49]  Peter Hellyer,et al.  A pilot investigation of the physical and psychological benefits of playing Pokémon GO for dog owners , 2017, Comput. Hum. Behav..

[50]  Limsoon Wong,et al.  Accomplishments and challenges in literature data mining for biology , 2002, Bioinform..

[51]  A. Apter.  Understanding adherence requires pragmatic trials: lessons from pediatric asthma. , 2015, JAMA pediatrics.

[52]  Iñaki Soto Rey,et al.  Automated Transformation of CDISC ODM to OpenClinica , 2017, GMDS.

[53]  Ross M Tonkens.  Breaking into clinical research. , 2005, Physician executive.

[54]  Ian Harvey,et al.  A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers , 2009, Canadian Medical Association Journal.

[55]  Philip T. Kortum,et al.  Psychometric Evaluation of the USE (Usefulness, Satisfaction, and Ease of use) Questionnaire for Reliability and Validity , 2018, Proceedings of the Human Factors and Ergonomics Society Annual Meeting.

[56]  Jettie Hoonhout,et al.  Design and baseline characteristics of the Food4Me study: a web-based randomised controlled trial of personalised nutrition in seven European countries , 2014, Genes & Nutrition.

[57]  J. S. Katz,et al.  What is research collaboration , 1997 .

[58]  Daniel Hind,et al.  Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme , 2017, BMJ Open.

[59]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[60]  A. Kesselheim,et al.  U.S. Food and Drug Administration Precertification Pilot Program for Digital Health Software: Weighing the Benefits and Risks , 2018, Annals of Internal Medicine.

[61]  P. Rothwell,et al.  Factors That Can Affect the External Validity of Randomised Controlled Trials , 2006, PLoS clinical trials.

[62]  Junseok Park,et al.  CORUS: Blockchain-Based Trustworthy Evaluation System for Efficacy of Healthcare Remedies , 2018, 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom).

[63]  L. Portney,et al.  Foundations of Clinical Research: Applications to Practice , 2015 .

[64]  M. Hogan,et al.  A protocol for a randomised active-controlled trial to evaluate the effects of an online mindfulness intervention on executive control, critical thinking and key thinking dispositions in a university student sample , 2016, BMC psychology.

[65]  L. F. Barrett,et al.  BEING EMOTIONAL DURING DECISION MAKING-GOOD OR BAD? AN EMPIRICAL INVESTIGATION. , 2007, Academy of Management journal. Academy of Management.

[66]  L. Kruger,et al.  Getting to Know Ourselves and Our Places Through Participation in Civic Social Assessment , 2000 .

[67]  Walid Maalej,et al.  On the automatic classification of app reviews , 2016, Requirements Engineering.

[68]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[69]  Robert J. Winter,et al.  Agile Software Development: Principles, Patterns, and Practices , 2014 .

[70]  J. Braverman,et al.  Measurement of Quality of Life in Patients with Mycosis Fungoides/Sézary Syndrome Cutaneous T-Cell Lymphoma: Development of an Electronic Instrument , 2019, Journal of medical Internet research.

[71]  Jelke Bethlehem,et al.  Selection Bias in Web Surveys , 2010 .

[72]  D. Paulhus,et al.  The over-claiming technique: measuring self-enhancement independent of ability. , 2003, Journal of personality and social psychology.

[73]  Arjen P. de Vries,et al.  Increasing cheat robustness of crowdsourcing tasks , 2013, Information Retrieval.

[74]  Brigitte Walther,et al.  Comparison of Electronic Data Capture (EDC) with the Standard Data Capture Method for Clinical Trial Data , 2011, PloS one.

[75]  J. Frost,et al.  Sharing Health Data for Better Outcomes on PatientsLikeMe , 2010, Journal of medical Internet research.

[76]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[77]  W. Hoeffding,et al.  Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling. , 1962 .

[78]  Jaeho Lee,et al.  Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications. , 2016, Issue brief.

[79]  Rosalind W. Picard,et al.  Use of In-Game Rewards to Motivate Daily Self-Report Compliance: Randomized Controlled Trial , 2019, Journal of medical Internet research.

[80]  Camarin E. Rolle,et al.  A pilot study to determine the feasibility of enhancing cognitive abilities in children with sensory processing dysfunction , 2017, PloS one.

[81]  Junseok Park,et al.  Concept embedding to measure semantic relatedness for biomedical information ontologies , 2019, J. Biomed. Informatics.

[82]  R. Califf,et al.  Real-World Evidence - What Is It and What Can It Tell Us? , 2016, The New England journal of medicine.

[83]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[84]  Daniel Smilkov,et al.  Similar image search for histopathology: SMILY , 2019, npj Digital Medicine.

[85]  Qian Xu,et al.  The bandwagon effect of collaborative filtering technology , 2008, CHI Extended Abstracts.

[86]  Jesse J. Chandler,et al.  Inside the Turk , 2014 .

[87]  M. Hogan,et al.  A randomised active-controlled trial to examine the effects of an online mindfulness intervention on executive control, critical thinking and key thinking dispositions in a university student sample , 2018, BMC psychology.

[88]  S. Kollins,et al.  Proof-of-concept study of an at-home, engaging, digital intervention for pediatric ADHD , 2018, PloS one.

[89]  James A. Pershing,et al.  Be a Performance Improvement Contributor: Write an Article! , 2014 .

[90]  Ralucca Gera,et al.  A heuristic approach to estimate nodes’ closeness rank using the properties of real world networks , 2018, Social Network Analysis and Mining.

[91]  J. Kitzinger,et al.  Participant Anonymity in the Internet Age: From Theory to Practice , 2015, Qualitative research in psychology.

[92]  R. Jamison,et al.  Electronic diaries for monitoring chronic pain: 1-year validation study , 2001, Pain.

[93]  R. Dellavalle,et al.  Mobile medical and health apps: state of the art, concerns, regulatory control and certification , 2014, Online journal of public health informatics.

[94]  Sanchita Bhattacharya,et al.  Prototype of running clinical trials in an untrustworthy environment using blockchain , 2019, Nature Communications.

[95]  John A. Johnson.  Ascertaining the validity of individual protocols from Web-based personality inventories. , 2005 .

[96]  D. Shepard,et al.  Financial Incentives to Increase Cardiac Rehabilitation Participation Among Low-Socioeconomic Status Patients: A Randomized Clinical Trial. , 2019, JACC. Heart failure.

[97]  Marc Buyse,et al.  Data fraud in clinical trials. , 2015, Clinical investigation.

[98]  E. Waltz.  Pear approval signals FDA readiness for digital treatments , 2018, Nature Biotechnology.

[99]  Robert L. Wolpert,et al.  Statistical Inference , 2019, Encyclopedia of Social Network Analysis and Mining.

[100]  H. Sox,et al.  Pragmatic Trials: Practical Answers to "Real World" Questions. , 2016, JAMA.

[101]  Ana Sofia Figueiredo,et al.  Data Sharing: Convert Challenges into Opportunities , 2017, Front. Public Health.

[102]  F. Galton.  Regression Towards Mediocrity in Hereditary Stature. , 1886 .

[103]  Dorian Pyle,et al.  Data Preparation for Data Mining , 1999 .

[104]  Adam B. Cohen,et al.  Digital health: a path to validation , 2019, npj Digital Medicine.

[105]  S Audrey,et al.  An informal school-based peer-led intervention for smoking prevention in adolescence (ASSIST): a cluster randomised trial , 2008, The Lancet.

[106]  J. Gordon,et al.  Online Recruitment Methods for Web-Based and Mobile Health Studies: A Review of the Literature , 2015, Journal of medical Internet research.