Two-Stream Convolutional Neural Network for Action Recognition in Radar

In the domain of human action classification, information obtained by optical equipment is used as the input source in most case. However, these optical devices are often affected by environmental factors, which bring some trouble to human motion recognition. The radar signal can avoid these situations, therefore the radar signal has been more and more significant part in this domain. In recent research, most researchers used the Micro-Doppler spectrogram of radar to classify the action. It only makes use of the time-frequency domain information, but ignores the spatial domain information. However, for the radar signal of the human motion, there should be some differences between the information in the time-frequency domain and the information in the other domain such as the space domain. In this case, we employed a CNN with two streams, which can use both Micro-Doppler spectrogram and the echo of the radar to recognize the action. In our architecture, there are two separate CNN processing streams for extracting the features of two stream of input, and the features are combined with a late network. The experiment proves that our method has better accuracy than the input of a single stream.

[1]  Youngwook Kim,et al.  Human Detection and Activity Classification Based on Micro-Doppler Signatures Using Deep Convolutional Neural Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[2]  Dave Tahmoush,et al.  Radar micro-doppler for long range front-view gait recognition , 2009, 2009 IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems.

[3]  Youngwook Kim,et al.  Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[5]  Xiaojun Jing,et al.  Deep Learning based Human Activity Classification in Radar Micro-Doppler Image , 2018, 2018 15th European Radar Conference (EuRAD).

[6]  Jeffrey A. Nanzer,et al.  Microwave and Millimeter-Wave Remote Sensing for Security Applications , 2020 .

[7]  Yuan He,et al.  Range-Doppler surface: a tool to analyse human target in ultra-wideband radar , 2015 .

[8]  Wolfram Burgard,et al.  Multimodal deep learning for robust RGB-D object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Ljubisa Stankovic,et al.  Analysis of radar micro-Doppler signatures from experimental helicopter and human data , 2007 .

[10]  Tian Jin,et al.  Transfer Learning for Human Activities Classification Using Micro-Doppler Spectrograms , 2018, 2018 IEEE International Conference on Computational Electromagnetics (ICCEM).

[11]  Cagatay Candan,et al.  Classification of human motion using radar micro-Doppler signatures with hidden Markov models , 2016, 2016 IEEE Radar Conference (RadarConf).

[12]  Pavan K. Turaga,et al.  Direct inference on compressive measurements using convolutional neural networks , 2016, 2016 IEEE International Conference on Image Processing (ICIP).