Latent HyperNet: Exploring the Layers of Convolutional Neural Networks

Since Convolutional Neural Networks (ConvNets) are able to simultaneously learn features and classifiers to discriminate different categories of activities, recent works have employed ConvNets approaches to perform human activity recognition (HAR) based on wearable sensors, allowing the removal of expensive human work and expert knowledge. However, these approaches have their power of discrimination limited mainly by the large number of parameters that compose the network and the reduced number of samples available for training. Inspired by this, we propose an accurate and robust approach, referred to as Latent HyperNet (LHN). The LHN uses feature maps from early layers (hyper) and projects them, individually, onto a low dimensionality (latent) space. Then, these latent features are concatenated and presented to a classifier. To demonstrate the robustness and accuracy of the LHN, we evaluate it using four different network architectures in five publicly available HAR datasets based on wearable sensors, which vary in the sampling rate and number of activities. We experimentally demonstrate that the proposed LHN is able to capture rich information, improving the results regarding the original ConvNets. Furthermore, the method outperforms existing state-of-the-art methods, on average, by 5.1 percentage points.

[1]  Mi Zhang,et al.  USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors , 2012, UbiComp.

[2]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[3]  Cagatay Catal,et al.  On the use of ensemble of classifiers for accelerometer-based activity recognition , 2015, Appl. Soft Comput..

[4]  Karthikeyan Natesan Ramamurthy,et al.  A deep learning approach to multiple kernel fusion , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  H. Abdi.  Partial least squares regression and projection on latent structure regression (PLS Regression) , 2010 .

[6]  Xianbin Wang,et al.  Mitigating sensor differences for phone-based human activity recognition , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[7]  William Robson Schwartz,et al.  Boosted Projection: An Ensemble of Transformation Models , 2017, CIARP.

[8]  Fuchun Sun,et al.  HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yuqing Chen,et al.  A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[10]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Kavita Bala,et al.  Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[13]  William Robson Schwartz,et al.  Learning to hash faces using large feature vectors , 2015, 2015 13th International Workshop on Content-Based Multimedia Indexing (CBMI).

[14]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Panagiotis Tsakalides,et al.  An online feature selection architecture for Human Activity Recognition , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[16]  Nasser Kehtarnavaz,et al.  UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[17]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[18]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[19]  Fulvio Mastrogiovanni,et al.  Wearable Inertial Sensors: Applications, Challenges, and Public Test Benches , 2015, IEEE Robotics & Automation Magazine.

[20]  Dan Morris,et al.  RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises , 2014, CHI.

[21]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[22]  Gary M. Weiss,et al.  Design considerations for the WISDM smart phone-based sensor mining architecture , 2011, SensorKDD '11.

[23]  Paul J. M. Havinga,et al.  A Survey of Online Activity Recognition Using Mobile Phones , 2015, Sensors.

[24]  Seungjin Choi,et al.  Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[25]  Héctor Pomares,et al.  mHealthDroid: A Novel Framework for Agile Development of Mobile Health Applications , 2014, IWAAL.

[26]  Larry S. Davis,et al.  Human detection using partial least squares analysis , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[27]  Juan C. Quiroz,et al.  Emotion-recognition using smart watch accelerometer data: preliminary findings , 2017, UbiComp/ISWC Adjunct.

[28]  Seungjin Choi,et al.  Multi-modal Convolutional Neural Networks for Activity Recognition , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[29]  Subhas Chandra Mukhopadhyay,et al.  Wearable Sensors for Human Activity Monitoring: A Review , 2015, IEEE Sensors Journal.

[30]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.