Multi-Feature Transformer-Based Learning for Continuous Human Motion Recognition with High Similarity Using mmWave FMCW Radar

Doppler-radar-based continuous human motion recognition recently has attracted extensive attention, which is a favorable choice for privacy and personal security. Existing results of continuous human motion recognition (CHMR) using mmWave FMCW Radar are not considered the continuous human motion with the high similarity problem. In this paper, we proposed a new CHMR algorithm with the consideration of the high similarity (HS) problem, called as CHMR-HS, by using the modified Transformer-based learning model. As far as we know, this is the first result in the literature to investigate the continuous HMR with the high similarity. To obtain the clear FMCW radar images, the background and target signals of the detected human are separated through the background denoising and the target extraction algorithms. To investigate the effects of the spectral-temporal multi-features with different dimensions, Doppler, range, and angle signatures are extracted as the 2D features and range-Doppler-time and range-angle-time signatures are extracted as the 3D features. The 2D/3D features are trained into the adjusted Transformer-encoder model to distinguish the difference of the high-similarity human motions. The conventional Transformer-decoder model is also re-designed to be Transformer-sequential-decoder model such that Transformer-sequential-decoder model can successfully recognize the continuous human motions with the high similarity. The experimental results show that the accuracy of our proposed CHMR-HS scheme are 95.2% and 94.5% if using 3D and 2D features, the simulation results also illustrates that our CHMR-HS scheme has advantages over existing CHMR schemes.

[1]  A. Temizel,et al.  Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation , 2022, MIUA.

[2]  Jun-Young Park,et al.  FMCW Radar Sensor Based Human Activity Recognition using Deep Learning , 2022, 2022 International Conference on Electronics, Information, and Communication (ICEIC).

[3]  Jianping Wang,et al.  CFAR-Based Interference Mitigation for FMCW Automotive Radar Systems , 2021, IEEE Transactions on Intelligent Transportation Systems.

[4]  A. Bourdoux,et al.  Multi-View CNN-LSTM Architecture for Radar-Based Human Activity Recognition , 2022, IEEE Access.

[5]  Xichan Zhu,et al.  Dynamic Hand Gesture Recognition in In-Vehicle Environment Based on FMCW Radar and Transformer , 2021, Sensors.

[6]  Akram Alomainy,et al.  Discrete Human Activity Recognition and Fall Detection by Combining FMCW RADAR Data of Heterogeneous Environments for Independent Assistive Living , 2021, Electronics.

[7]  Xiaojun Jing,et al.  Human Motion Recognition With Limited Radar Micro-Doppler Signatures , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Orhan Arikan,et al.  Capon's Beamformer and Minimum Mean Square Error Beamforming Techniques in Direction of Arrival Estimation , 2021, 2021 29th Signal Processing and Communications Applications Conference (SIU).

[9]  A. Yarovoy,et al.  Continuous human activity recognition for arbitrary directions with distributed radars , 2021, 2021 IEEE Radar Conference (RadarConf21).

[10]  Tim Fingscheidt,et al.  Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition , 2021, Interspeech.

[11]  Jian Liu,et al.  An End-to-End Network for Continuous Human Motion Recognition via Radar Radios , 2021, IEEE Sensors Journal.

[12]  Ronghang Hu,et al.  UniT: Multimodal Multitask Learning with a Unified Transformer , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Xinhua Liu,et al.  Continuous Human Motion Recognition Using Micro-Doppler Signatures in the Scenario With Micro Motion Interference , 2021, IEEE Sensors Journal.

[14]  S. Muramatsu,et al.  Accuracy Improvement of Human Motion Recognition with MW-FMCW Radar Using CNN , 2021, 2020 International Symposium on Antennas and Propagation (ISAP).

[15]  Hector A. Gonzalez,et al.  Phase-based Doppler Disambiguation in TDM and BPM MIMO FMCW Radars , 2021, 2021 IEEE Radio and Wireless Symposium (RWS).

[16]  Vasileios Papanastasiou,et al.  Deep Learning-based identification of human gait by radar micro-Doppler measurements , 2021, 2020 17th European Radar Conference (EuRAD).

[17]  Francesco Fioranelli,et al.  Continuous Human Activity Classification From FMCW Radar With Bi-LSTM Networks , 2020, IEEE Sensors Journal.

[18]  Shih-Hau Fang,et al.  The Impact of Environmental Factors on mm-Wave Radar Point-Clouds for Human Activity Recognition , 2020, 2020 International Workshop on Electromagnetics: Applications and Student Innovation Competition (iWEM).

[19]  Yilong Lu,et al.  Classification and Identification of Micro-Doppler Signatures of Pedestrian and Cyclists Using Simulated Data and ELM , 2019, 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP).

[20]  Xiaohua Zhu,et al.  Continuous Human Motion Recognition With a Dynamic Range-Doppler Trajectory Method Based on FMCW Radar , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[21]  Kevin Chetty,et al.  On the Application of Digital Moving Target Indication Techniques to Short-Range FMCW Radar Data , 2018, IEEE Sensors Journal.

[22]  Mu Zhou,et al.  Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor , 2018, IEEE Sensors Journal.

[23]  Yuliang Sun,et al.  Gesture Classification with Handcrafted Micro-Doppler Features using a FMCW Radar , 2018, 2018 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM).

[24]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[25]  Youngwook Kim,et al.  Human Detection and Activity Classification Based on Micro-Doppler Signatures Using Deep Convolutional Neural Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[26]  Jürgen Schmidhuber,et al.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks , 2006, ICML.

[27]  Victor C. Chen,et al.  Analysis of radar micro-Doppler with time-frequency transform , 2000, Proceedings of the Tenth IEEE Workshop on Statistical Signal and Array Processing (Cat. No.00TH8496).