Moving the Lab into the Mountains: A Pilot Study of Human Activity Recognition in Unstructured Environments

Goal: To develop and validate a field-based data collection and assessment method for human activity recognition in the mountains with variations in terrain and fatigue using a single accelerometer and a deep learning model. Methods: The protocol generated an unsupervised labelled dataset of various long-term field-based activities including run, walk, stand, lay and obstacle climb. Activity was voluntary so transitions could not be determined a priori. Terrain variations included slope, crossing rivers, obstacles and surfaces including road, gravel, clay, mud, long grass and rough track. Fatigue levels were modulated between rested to physical exhaustion. The dataset was used to train a deep learning convolutional neural network (CNN) capable of being deployed on battery powered devices. The human activity recognition results were compared to a lab-based dataset with 1,098,204 samples and six features, uniform smooth surfaces, non-fatigued supervised participants and activity labelling defined by the protocol. Results: The trail run dataset had 3,829,759 samples with five features. The repetitive activities and single instance activities required hyper parameter tuning to reach an overall accuracy 0.978 with a minimum class precision for the one-off activity (climbing gate) of 0.802. Conclusion: The experimental results showed that the CNN deep learning model performed well with terrain and fatigue variations compared to the lab equivalents (accuracy 97.8% vs. 97.7% for trail vs. lab). Significance: To the authors knowledge this study demonstrated the first successful human activity recognition (HAR) in a mountain environment. A robust and repeatable protocol was developed to generate a validated trail running dataset when there were no observers present and activity types changed on a voluntary basis across variations in terrain surface and both cognitive and physical fatigue levels.

[1]  Brian Caulfield,et al.  Binary classification of running fatigue using a single inertial measurement unit , 2017, 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[2]  Cheng Xu,et al.  InnoHAR: A Deep Neural Network for Complex Human Activity Recognition , 2019, IEEE Access.

[3]  Thaier Hayajneh,et al.  Smartphone and Smartwatch-Based Biometrics Using Activities of Daily Living , 2019, IEEE Access.

[4]  Michael Goldfarb,et al.  A Phase Variable Approach for IMU-Based Locomotion Activity Recognition , 2018, IEEE Transactions on Biomedical Engineering.

[5]  Qingquan Li,et al.  Impact of Sliding Window Length in Indoor Human Motion Modes and Pose Pattern Recognition Based on Smartphone Sensors , 2018, Sensors.

[6]  J. Caldwell,et al.  The Effects of 37 Hours of Continuous Wakefulness On the Physiological Arousal, Cognitive Performance, Self-Reported Mood, and Simulator Flight Performance of F-117A Pilots , 2004 .

[7]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[8]  Davide Anguita,et al.  Human Activity Recognition on Smartphones Using a Multiclass Hardware-Friendly Support Vector Machine , 2012, IWAAL.

[9]  Héctor Pomares,et al.  Window Size Impact in Human Activity Recognition , 2014, Sensors.

[10]  L. V. D. van der Woude,et al.  Recommended Measures for the Assessment of Cognitive and Physical Performance in Older Patients with Dementia: A Systematic Review , 2012, Dementia and Geriatric Cognitive Disorders Extra.

[11]  Dominic Thewlis,et al.  Tracking Performance Changes With Running-Stride Variability When Athletes Are Functionally Overreached. , 2017, International journal of sports physiology and performance.

[12]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[13]  T. Watson,et al.  Bioharness(™) Multivariable Monitoring Device: Part. II: Reliability. , 2012, Journal of sports science & medicine.

[14]  Gerwyn Hughes,et al.  Bioharness(™) multivariable monitoring device: part. I: validity. , 2012, Journal of sports science & medicine.

[15]  Hwee Pink Tan,et al.  Deep Activity Recognition Models with Triaxial Accelerometers , 2015, AAAI Workshop: Artificial Intelligence Applied to Assistive Technologies and Smart Environments.

[16]  Joaquín B. Ordieres Meré,et al.  Comparison of Data Preprocessing Approaches for Applying Deep Learning to Human Activity Recognition in the Context of Industry 4.0 , 2018, Sensors.

[17]  Roozbeh Jafari,et al.  Personalizing Activity Recognition Models Through Quantifying Different Types of Uncertainty Using Wearable Sensors , 2020, IEEE Transactions on Biomedical Engineering.

[18]  Suchetana Chakraborty,et al.  Activity Recognition through Feature Learning and Annotations using LSTM , 2019, 2019 11th International Conference on Communication Systems & Networks (COMSNETS).

[19]  Dinesh Kumar Vishwakarma,et al.  A review of state-of-the-art techniques for abnormal human activity recognition , 2019, Eng. Appl. Artif. Intell..

[20]  S. Miyake,et al.  Physiological responses to workload change. A test/retest examination. , 2009, Applied ergonomics.

[21]  Ajmal Mian,et al.  Predicting Athlete Ground Reaction Forces and Moments From Spatio-Temporal Driven CNN Models , 2019, IEEE Transactions on Biomedical Engineering.

[22]  S. J. Sullivan,et al.  Reliability and diagnostic characteristics of clinical tests of upper limb motor function , 2012, Journal of Clinical Neuroscience.

[23]  Koushik Maharatna,et al.  Rehab-Net: Deep Learning Framework for Arm Movement Classification Using Wearable Sensors for Stroke Rehabilitation , 2019, IEEE Transactions on Biomedical Engineering.

[24]  Susan R. Homack,et al.  A meta-analysis of the sensitivity and specificity of the Stroop Color and Word Test with children. , 2004, Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists.

[25]  Juha Röning,et al.  Clustering-based activity classification with a wrist-worn accelerometer using basic features , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[26]  Yu Wang,et al.  An Automatic User-Adapted Physical Activity Classification Method Using Smartphones , 2017, IEEE Transactions on Biomedical Engineering.

[27]  Ying Wah Teh,et al.  Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges , 2018, Expert Syst. Appl..

[28]  Andrey Ignatov,et al.  Real-time human activity recognition from accelerometer data using Convolutional Neural Networks , 2018, Appl. Soft Comput..

[29]  G. Borg Psychophysical bases of perceived exertion. , 1982, Medicine and science in sports and exercise.

[30]  Anantha Narayanan,et al.  Application of Raw Accelerometer Data and Machine-Learning Techniques to Characterize Human Movement Behavior: A Systematic Scoping Review. , 2020, Journal of physical activity & health.

[31]  Nicholas G. Murray,et al.  Athletes who train on unstable compared to stable surfaces exhibit unique postural control strategies in response to balance perturbations , 2016, Journal of sport and health science.

[32]  Stephen J. McKenna,et al.  Computer Vision and Image Understanding Recognising Complex Activities with Histograms of Relative Tracklets , 2022 .

[33]  Yoshihito Shigihara,et al.  Two different types of mental fatigue produce different styles of task performance , 2013 .

[34]  Teh Ying Wah,et al.  Data fusion and multiple classifier systems for human activity detection and health monitoring: Review and open research directions , 2019, Inf. Fusion.

[35]  Sam Robertson,et al.  Machine and deep learning for sport-specific movement recognition: a systematic review of model development and performance , 2018, Journal of sports sciences.

[36]  Yuqing Chen,et al.  A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[37]  Lifen Jiang,et al.  HDL: Hierarchical Deep Learning Model based Human Activity Recognition using Smartphone Sensors , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).