The Importance of Real-World Validation of Machine Learning Systems in Wearable Exercise Biofeedback Platforms: A Case Study

Machine learning models are being utilized to provide wearable sensor-based exercise biofeedback to patients undertaking physical therapy. However, most systems are validated at a technical level using lab-based cross validation approaches. These results do not necessarily reflect the performance levels that patients and clinicians can expect in the real-world environment. This study aimed to conduct a thorough evaluation of an example wearable exercise biofeedback system from laboratory testing through to clinical validation in the target setting, illustrating the importance of context when validating such systems. Each of the various components of the system were evaluated independently, and then in combination as the system is designed to be deployed. The results show a reduction in overall system accuracy between lab-based cross validation (>94%), testing on healthy participants (n = 10) in the target setting (>75%), through to test data collected from the clinical cohort (n = 11) (>59%). This study illustrates that the reliance on lab-based validation approaches may be misleading key stakeholders in the inertial sensor-based exercise biofeedback sector, makes recommendations for clinicians, developers and researchers, and discusses factors that may influence system performance at each stage of evaluation.

[1]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[2]  Simon Hunter,et al.  Translating Comprehensive Conservative Care for Chronic Knee Pain Into a Digital Care Pathway: 12-Week and 6-Month Outcomes for the Hinge Health Program , 2017, JMIR rehabilitation and assistive technologies.

[3]  Cynthia S Crowson,et al.  Prevalence of Total Hip and Knee Replacement in the United States. , 2015, The Journal of bone and joint surgery. American volume.

[4]  Kenneth Meijer,et al.  Activity identification using body-mounted sensors—a review of classification techniques , 2009, Physiological measurement.

[5]  Brian Caulfield,et al.  Rehabilitation exercise assessment using inertial sensors: a cross-sectional analytical study , 2014, Journal of NeuroEngineering and Rehabilitation.

[6]  Madalina Fiterau,et al.  Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities. , 2018, Journal of biomechanics.

[7]  Brian Caulfield,et al.  Wearable Sensor-Based Exercise Biofeedback for Orthopaedic Rehabilitation: A Mixed Methods User Evaluation of a Prototype System , 2019, Sensors.

[8]  Takeo Kanade,et al.  Multi-label classification for the analysis of human motion quality , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  Diogo R. Ferreira,et al.  Preprocessing techniques for context recognition from accelerometer data , 2010, Personal and Ubiquitous Computing.

[10]  C.-C. Jay Kuo,et al.  Audio content analysis for online audiovisual data segmentation and classification , 2001, IEEE Trans. Speech Audio Process..

[11]  Simon Hunter,et al.  Randomized controlled trial of a 12-week digital care program in improving low back pain , 2019, npj Digital Medicine.

[12]  Oonagh M. Giggins,et al.  Biofeedback in rehabilitation , 2013, Journal of NeuroEngineering and Rehabilitation.

[13]  Darragh F Whelan,et al.  Technology in Strength and Conditioning: Assessing Bodyweight Squat Technique With Wearable Sensors. , 2017, Journal of strength and conditioning research.

[14]  Nigel H. Lovell,et al.  Review: Are we stumbling in our quest to find the best predictor? Over-optimism in sensor-based models for predicting falls in older adults , 2015, Healthcare technology letters.

[15]  William Johnston,et al.  Wearable Inertial Sensor Systems for Lower Limb Exercise Detection and Evaluation: A Systematic Review , 2018, Sports Medicine.

[16]  Christopher Tack,et al.  Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy. , 2019, Musculoskeletal science & practice.

[17]  Gustavo E. A. P. A. Batista,et al.  A study of the behavior of several methods for balancing machine learning training data , 2004, SKDD.

[18]  M. Tahar Kechadi,et al.  Leveraging IMU data for accurate exercise performance classification and musculoskeletal injury risk screening , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  Jonathan Feng-Shun Lin,et al.  Classification-based Segmentation for Rehabilitation Exercise Monitoring , 2018, Journal of rehabilitation and assistive technologies engineering.

[20]  Dana Kulic,et al.  Online Segmentation and Clustering From Continuous Observation of Whole Body Motions , 2009, IEEE Transactions on Robotics.

[21]  S. Mawson,et al.  Telerehabilitation: enabling the remote delivery of healthcare, rehabilitation, and self management. , 2009, Studies in health technology and informatics.

[22]  Darragh F Whelan,et al.  Determining Interrater and Intrarater Levels of Agreement in Students and Clinicians When Visually Evaluating Movement Proficiency During Screening Assessments , 2019, Physical therapy.

[23]  M. Tahar Kechadi,et al.  Automatic classification of knee rehabilitation exercises using a single inertial sensor: A case study , 2018, 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[24]  Brian Caulfield,et al.  A Novel Validation Framework to Assess Segmentation Accuracy of Inertial Sensor Data for Rehabilitation Exercises , 2020 .

[25]  Jonathan Feng-Shun Lin,et al.  Online Segmentation of Human Motion for Automated Rehabilitation Exercise Analysis , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[26]  P. Lehoux,et al.  A systematic review of clinical outcomes, clinical process, healthcare utilization and costs associated with telerehabilitation , 2009, Disability and rehabilitation.

[27]  Michelle Karg,et al.  Movement Primitive Segmentation for Human Motion Modeling: A Framework for Analysis , 2016, IEEE Transactions on Human-Machine Systems.

[28]  Brian Caulfield,et al.  Mobile App to Streamline the Development of Wearable Sensor-Based Exercise Biofeedback Systems: System Development and Evaluation , 2017, JMIR rehabilitation and assistive technologies.

[29]  M. Tahar Kechadi,et al.  The limb movement analysis of rehabilitation exercises using wearable inertial sensors , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[30]  Adrian Burns,et al.  SHIMMER™ – A Wireless Sensor Platform for Noninvasive Biomedical Research , 2010, IEEE Sensors Journal.

[31]  Brian Caulfield,et al.  Classification of lunge biomechanics with multiple and individual inertial measurement units , 2017, Sports biomechanics.

[32]  Kun-Hui Chen,et al.  Wearable Sensor-Based Rehabilitation Exercise Assessment for Knee Osteoarthritis , 2015, Sensors.

[33]  M. Tahar Kechadi,et al.  Combining Real-Time Segmentation and Classification of Rehabilitation Exercises with LSTM Networks and Pointwise Boosting , 2020, AAAI.