Activity Recognition from Multi-modal Sensor Data Using a Deep Convolutional Neural Network

Multi-modal data extracted from different sensors in a smart home can be fused to build models that recognize the daily living activities of residents. This paper proposes a Deep Convolutional Neural Network to perform the activity recognition task using the multi-modal data collected from a smart residential home. The dataset contains accelerometer data (composed of three perpendicular components of acceleration and the strength of the accelerometer signal received by four receivers), video data (15 time-series related to 2D and 3D center of mass and bounding box extracted from an RGB-D camera), and Passive Infra-Red sensor data. The performance of the Deep Convolutional Neural Network is compared to the Deep Belief Network. Experimental results revealed that the Deep Convolutional Neural Network with two pairs of convolutional and max pooling layers achieved better classification accuracy than the Deep Belief Network. The Deep Belief Network uses Restricted Boltzmann Machines for pre-training the network. When training deep learning models using classes with a high number of training samples, the DBN achieved 65.97% classification accuracy, whereas the CNN achieved 75.33% accuracy. The experimental results demonstrate the challenges of dealing with multi-modal data and highlight the importance of having a suitable number of samples within each class for sufficiently training and testing deep learning models.

[1]  Xiaohong W. Gao,et al.  Classification of CT brain images based on deep learning networks , 2017, Comput. Methods Programs Biomed..

[2]  Xiaola Lin,et al.  Feature extraction using Restricted Boltzmann Machine for stock price prediction , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE).

[3]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[4]  Niall Twomey,et al.  Unsupervised learning of sensor topologies for improving activity recognition in smart environments , 2017, Neurocomputing.

[5]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Niall Twomey,et al.  Bayesian Active Learning with Evidence-Based Instance Selection , 2015 .

[8]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[9]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[10]  Kunihiko Fukushima,et al.  Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position , 1982, Pattern Recognit..

[11]  Niall Twomey,et al.  The SPHERE Challenge: Activity Recognition with Multimodal Sensor Data , 2016, ArXiv.

[12]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[13]  Haipeng Wang,et al.  Target Classification Using the Deep Convolutional Networks for SAR Images , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Alberto Signoroni,et al.  Bacterial colony counting with Convolutional Neural Networks in Digital Microbiology Imaging , 2017, Pattern Recognit..

[15]  Luiz Eduardo Soares de Oliveira,et al.  Breast cancer histopathological image classification using Convolutional Neural Networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[16]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[17]  Zhenjiang Miao,et al.  Feature extraction with convolutional restricted boltzmann machine for audio classification , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[18]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[19]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[20]  Jonathan Cheung-Wai Chan,et al.  Hyperspectral image classification using two-channel deep convolutional neural network , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).