Human Activity Recognition Using 3D Orthogonally-projected EfficientNet on Radar Time-Range-Doppler Signature

In radar activity recognition, 2D signal representations such as spectrogram, cepstrum and cadence velocity diagram are often utilized, while range information is often neglected. In this work, we propose to utilize the 3D time-range-Doppler (TRD) representation, and design a 3D Orthogonally-Projected EfficientNet (3D-OPEN) to effectively capture the discriminant information embedded in the 3D TRD cubes for accurate classification. The proposed model aggregates the discriminant information from three orthogonal planes projected from the 3D feature space. It alleviates the difficulty of 3D CNNs in exploiting sparse semantic abstractions directly from the high-dimensional 3D representation. The proposed method is evaluated on the Millimeter-Wave Radar Walking Dataset. It significantly and consistently outperforms the state-of-the-art methods for radar activity recognition.

[1]  M. Bernardi,et al.  Temporal Convolutional Neural Networks for Radar Micro-Doppler Based Gait Recognition † , 2021, Sensors.

[2]  Alexandros Iosifidis,et al.  Minimum Variance Extreme Learning Machine for human action recognition , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Moeness G. Amin,et al.  Radar Data Cube Processing for Human Activity Recognition Using Multisubspace Learning , 2019, IEEE Transactions on Aerospace and Electronic Systems.

[4]  Hong Liu,et al.  Learning directional co-occurrence for human action classification , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  Ming Ye,et al.  Radar‐ID: human identification based on radar micro‐Doppler signatures using deep convolutional neural networks , 2018, IET Radar, Sonar & Navigation.

[6]  Xudong Jiang,et al.  Regularized 2-D complex-log spectral analysis and subspace reliability analysis of micro-Doppler signature for UAV detection , 2017, Pattern Recognit..

[7]  Wei Wang,et al.  Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[8]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[9]  Olivier Romain,et al.  Human activity classification with radar signal processing and machine learning , 2020, 2020 International Conference on UK-China Emerging Technologies (UCET).

[10]  Ennio Gambi,et al.  Millimeter wave radar data of people walking , 2020, Data in brief.

[11]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Chris Harrison,et al.  Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition , 2021, CHI.

[13]  Lingxiao He,et al.  Video-based Person Re-identification via 3D Convolutional Networks and Non-local Attention , 2018, ACCV.

[14]  Ivan Poupyrev,et al.  Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum , 2016, UIST.

[15]  Zhangjing Wang,et al.  Rapid Recognition of Human Behavior Based on Micro-Doppler Feature , 2019, 2019 International Conference on Control, Automation and Information Sciences (ICCAIS).

[16]  Shilin Zhu,et al.  Gait Recognition for Co-Existing Multiple People Using Millimeter Wave Sensing , 2020, AAAI.

[17]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[18]  Xinyu Li,et al.  A Survey of Deep Learning-Based Human Activity Recognition in Radar , 2019, Remote. Sens..

[19]  Pavlo Molchanov,et al.  Multi-sensor system for driver's hand-gesture recognition , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[20]  Ennio Gambi,et al.  People Walking Classification Using Automotive Radar , 2020, Electronics.

[21]  Xiaohua Zhu,et al.  Continuous Human Motion Recognition With a Dynamic Range-Doppler Trajectory Method Based on FMCW Radar , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[22]  Xudong Jiang,et al.  A three-step classification framework to handle complex data distribution for radar UAV detection , 2021, Pattern Recognit..

[23]  Yutaka Satoh,et al.  Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.