SensorNet: A Scalable and Low-Power Deep Convolutional Neural Network for Multimodal Data Classification

This paper presents SensorNet which is a scalable and low-power embedded deep convolutional neural network (DCNN), designed to classify multimodal time series signals. Time series signals generated by different sensor modalities with different sampling rates are first converted to images (2-D signals), and then DCNN is utilized to automatically learn shared features in the images and perform the classification. SensorNet: 1) is scalable as it can process different types of time series data with variety of input channels and sampling rates; 2) does not need to employ separate signal processing techniques for processing the data generated by each sensor modality; 3) does not require expert knowledge for extracting features for each sensor data; 4) makes it easy and fast to adapt to new sensor modalities with a different sampling rate; 5) achieves very high detection accuracy for different case studies; and 6) has a very efficient architecture which makes it suitable to be deployed at Internet of Things and wearable devices. A custom low-power hardware architecture is also designed for the efficient deployment of SensorNet at embedded real-time systems. SensorNet performance is evaluated using three different case studies including physical activity monitoring, stand-alone tongue drive system, and stress detection, and it achieves an average detection accuracy of 98%, 96.2%, and 94% for each case study, respectively. We implement SensorNet using our custom hardware architecture on Xilinx FPGA (Artix-7) which on average consumes 246-<inline-formula> <tex-math notation="LaTeX">$\mu \text{J}$ </tex-math></inline-formula> energy. To further reduce the power consumption, SensorNet is implemented using application-specified integrated circuit at the post layout level in 65-nm CMOS technology which consumes approximately <inline-formula> <tex-math notation="LaTeX">$8\times $ </tex-math></inline-formula> lower power compared to the FPGA implementation. In addition, SensorNet is implemented on NVIDIA Jetson TX2 SoC (CPU + GPU) and compared to TX2 single-core CPU and GPU implementations, FPGA-based SensorNet obtains <inline-formula> <tex-math notation="LaTeX">$15\times $ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> improvement in energy consumption.

[1]  Andrew Y. Ng,et al.  Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks , 2017, ArXiv.

[2]  Maysam Ghovanloo,et al.  A Low-Power Wearable Stand-Alone Tongue Drive System for People With Severe Disabilities , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[3]  Hui Ding,et al.  Querying and mining of time series data: experimental comparison of representations and distance measures , 2008, Proc. VLDB Endow..

[4]  Rajesh Gupta,et al.  Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs , 2017, FPGA.

[5]  Didier Stricker,et al.  Introducing a modular activity monitoring system , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  Shaohan Hu,et al.  DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing , 2016, WWW.

[7]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[8]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.

[9]  T. Mohsenin,et al.  SPARCNet , 2017 .

[10]  Soheil Ghiasi,et al.  Design space exploration of FPGA-based Deep Convolutional Neural Networks , 2016, 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC).

[11]  Sajal K. Das,et al.  A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[12]  Luca Benini,et al.  YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration , 2016, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[13]  George Manis,et al.  Heartbeat Time Series Classification With Support Vector Machines , 2009, IEEE Transactions on Information Technology in Biomedicine.

[14]  Mahesh K. Marina,et al.  Towards multimodal deep learning for activity recognition on mobile devices , 2016, UbiComp Adjunct.

[15]  Vivienne Sze,et al.  14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks , 2016, ISSCC.

[16]  Yi Zheng,et al.  Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks , 2014, WAIM.

[17]  Urbashi Mitra,et al.  Multimodal Physical Activity Recognition by Fusing Temporal and Cepstral Information , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[18]  Tinoosh Mohsenin,et al.  BiNMAC: Binarized neural Network Manycore ACcelerator , 2018, ACM Great Lakes Symposium on VLSI.

[19]  Ivan Marsic,et al.  Concurrent Activity Recognition with Multimodal CNN-LSTM Structure , 2017, ArXiv.

[20]  Tim Oates,et al.  Encoding Time Series as Images for Visual Inspection and Classification Using Tiled Convolutional Neural Networks , 2014 .

[21]  Frédéric Pétrot,et al.  Ternary neural networks for resource-efficient AI applications , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[22]  Javier Echanobe,et al.  A wearable human activity recognition system on a chip , 2014, Proceedings of the 2014 Conference on Design and Architectures for Signal and Image Processing.

[23]  Jason Cong,et al.  Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks , 2015, FPGA.

[24]  Wenyu Zhang,et al.  Multivariate Time Series Classification Using Dynamic Time Warping Template Selection for Human Activity Recognition , 2015, 2015 IEEE Symposium Series on Computational Intelligence.

[25]  Tinoosh Mohsenin,et al.  A low power seizure detection processor based on direct use of compressively-sensed data and employing a deterministic random matrix , 2015, 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[26]  Tsutomu Sasao,et al.  A deep convolutional neural network based on nested residue number system , 2015, 2015 25th International Conference on Field Programmable Logic and Applications (FPL).

[27]  Mehrdad Nourani,et al.  A Non-EEG Biosignals Dataset for Assessment and Visualization of Neurological Status , 2016, 2016 IEEE International Workshop on Signal Processing Systems (SiPS).

[28]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[29]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[30]  Zhongfeng Wang,et al.  An Energy-Efficient Architecture for Binary Weight Convolutional Neural Networks , 2018, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[31]  Maysam Ghovanloo,et al.  An embedded FPGA accelerator for a stand-alone dual-mode assistive device , 2017, 2017 IEEE Biomedical Circuits and Systems Conference (BioCAS).