Multi-attention deep recurrent neural network for nursing action evaluation using wearable sensor

A nursing action evaluation system that can assess the performance of students practicing patient handling related nursing skills becomes an urgent need for solving the nursing educator shortage problem. Such an evaluation system should be designed with less hand-crafted procedures for its scalability. Additionally, realizing high accuracy of nursing action recognition, especially fine-grained action recognition remains a problem. This reflects in the recognition of the correct and incorrect methods when students perform a nursing action, and low accuracy of that would mislead the nursing students. We propose a multi-attention deep recurrent neural network (MA-DRNN) model for nursing action recognition by directly processing the raw acceleration and rotational speed signals from wearable sensors. Data samples of target nursing actions in a nursing skill called patient transfer were collected to train and compare the models. The experiment results show that the proposed model can reach approximately 96% recognition accuracy for four target fine-grained nursing action classes helped by the attention mechanism on time and layer domains, which outperforms the state-of-the-art models of wearable sensor-based HAR.

[1]  Pmhcns-Bc Faan Deena A. Nardi PhD,et al.  The Global Nursing Faculty Shortage: Status and Solutions for Change , 2013 .

[2]  Yu Zhao,et al.  Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors , 2017, Mathematical Problems in Engineering.

[3]  Shinichi Furuya,et al.  Organization of the upper limb movement for piano key-depression differs between expert pianists and novice players , 2008, Experimental Brain Research.

[4]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[5]  Jochen Teizer,et al.  Real-time construction worker posture analysis for ergonomics training , 2012, Adv. Eng. Informatics.

[6]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[7]  H C Standish,et al.  Nursing shortage. , 2001, Connecticut medicine.

[8]  Ramakant Nevatia,et al.  Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images , 2015, ACM Multimedia.

[9]  Yonggang Wen,et al.  Multicolumn Bidirectional Long Short-Term Memory for Mobile Devices-Based Human Activity Recognition , 2016, IEEE Internet of Things Journal.

[10]  Cordelia Schmid,et al.  P-CNN: Pose-Based CNN Features for Action Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Lutz Prechelt,et al.  Early Stopping - But When? , 2012, Neural Networks: Tricks of the Trade.

[12]  A. Garg,et al.  An ergonomic evaluation of nursing assistants' job in a nursing home. , 1992, Ergonomics.

[13]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.

[14]  Yutaka Satoh,et al.  Fine-Grained Walking Activity Recognition via Driving Recorder Dataset , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[15]  A. Bartosiewicz,et al.  A nursing shortage – a prospect of global and local policies , 2018, International nursing review.

[16]  Li Deng,et al.  A tutorial survey of architectures, algorithms, and applications for deep learning , 2014, APSIPA Transactions on Signal and Information Processing.

[17]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[18]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[19]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[20]  Patricia A Grady,et al.  Advancing the health of our aging population: a lead role for nursing science. , 2011, Nursing outlook.

[21]  Ming Shao,et al.  A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Jun Ota,et al.  Translational Acceleration, Rotational Speed, and Joint Angle of Patients Related to Correct/Incorrect Methods of Transfer Skills by Nurses , 2018, Sensors.

[23]  S. Venkatesan,et al.  A Framework for Golf Training Using Low-Cost Inertial Sensors , 2010, 2010 International Conference on Body Sensor Networks.

[24]  Andrew A. West,et al.  Design and Implementation of an Integrated Performance Monitoring Tool for Swimming to Extract Stroke Information at Real Time , 2013, IEEE Transactions on Human-Machine Systems.

[25]  M. Gams,et al.  Comparing Deep and Classical Machine Learning Methods for Human Activity Recognition using Wrist Accelerometer , 2016 .

[26]  Bernt Schiele,et al.  A database for fine grained activity detection of cooking activities , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Makoto Hashizume,et al.  Waseda Bioinstrumentation system WB-3 as a wearable tool for objective laparoscopic skill evaluation , 2011, 2011 IEEE International Conference on Robotics and Automation.

[29]  Bingbing Ni,et al.  Interaction part mining: A mid-level approach for fine-grained action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[31]  Taku Komura,et al.  A Virtual Reality Dance Training System Using Motion Capture Technology , 2011, IEEE Transactions on Learning Technologies.

[32]  Björn W. Schuller,et al.  Robust discriminative keyword spotting for emotionally colored spontaneous speech using bidirectional LSTM networks , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[33]  Jae-Young Pyun,et al.  Deep Recurrent Neural Networks for Human Activity Recognition , 2017, Sensors.

[34]  Jun Ota,et al.  Automatic Evaluation of Trainee Nurses' Patient Transfer Skills Using Multiple Kinect Sensors , 2014, IEICE Trans. Inf. Syst..