High Fidelity Physics Simulation-Based Convolutional Neural Network for Automotive Radar Target Classification Using Micro-Doppler

Detection and classification of vulnerable road users (VRUs) such as pedestrians and cyclists is a key requirement for the realization of fully autonomous vehicles. Radar-based classification of VRUs can be achieved by exploiting differences in the micro-Doppler signatures associated with VRUs. Specifically, machine learning (ML) algorithms can be trained to classify VRUs using the spectral content of radar signals. The performance of these models depends on the quality and quantity of the data used during the training process. Currently, data collection is typically done through measurements or low fidelity physics, primitive-based simulations. The feasibility of carrying out measurements to collect training data is typically limited by the vast amounts of data required and practicality issues when using VRUs like animals. In this paper, we present a computationally efficient, high fidelity physics-based simulation workflow that can be used to obtain a large quantity of spectrograms from the micro-Doppler signatures of VRUs. The simulations are conducted on full-scale VRU models with a 77 GHz, frequency-modulated continuous-wave (FMCW) radar sensor model. Here, we collect the spectrograms of 4 targets; car, pedestrian, cyclist and dog at different speeds and angles-of-arrival. This data is then used to train a 5-layer convolutional neural network (CNN) that achieves nearly 100% classification accuracy after 5 epochs. Studies are conducted to investigate the impact of training data size, velocity and observation time window size on the accuracy of the CNN. Results from this study demonstrate how an accuracy of 95% can be realized using spectrograms obtained over a 0.2 s time window.

[1]  Xinyu Li,et al.  A Survey of Deep Learning-Based Human Activity Recognition in Radar , 2019, Remote. Sens..

[2]  Onur Toker,et al.  Design of a Cyberattack Resilient 77 GHz Automotive Radar Sensor , 2020, Electronics.

[3]  Kahlid Soofi,et al.  Image Classification , 2008, Encyclopedia of Multimedia.

[4]  Ali Cafer Gürbüz,et al.  Importance ranking of features for human micro-Doppler classification with a radar network , 2013, Proceedings of the 16th International Conference on Information Fusion.

[5]  Chi-Chih Chen,et al.  Micro-Doppler characteristics of pedestrians and bicycles for automotive radar sensors at 77 GHz , 2017, 2017 11th European Conference on Antennas and Propagation (EUCAP).

[6]  Hongwei Liu,et al.  Moving vehicle classification based on micro-Doppler signature , 2011, 2011 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC).

[7]  Jian Li,et al.  MIMO Radar with Colocated Antennas , 2007, IEEE Signal Processing Magazine.

[8]  Igal Bilik,et al.  Application of a 24 GHz FMCW automotive radar for urban target classification , 2014, 2014 IEEE Radar Conference.

[9]  C. J. Reddy,et al.  Near-Field RCS for Automotive Radar Applications , 2019, 2019 International Workshop on Antenna Technology (iWAT).

[10]  Ming Chen,et al.  77 GHz radar scattering properties of pedestrians , 2014, 2014 IEEE Radar Conference.

[11]  Janusz Szpytko,et al.  Safety problems in vehicles with adaptive cruise control system , 2017 .

[12]  Youngwook Kim,et al.  Human Detection and Activity Classification Based on Micro-Doppler Signatures Using Deep Convolutional Neural Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[13]  T. Zwick,et al.  Millimeter-Wave Technology for Automotive Radar Sensors in the 77 GHz Frequency Band , 2012, IEEE Transactions on Microwave Theory and Techniques.

[14]  Implementing Digital Processing for Automotive Radar Using SoCs , 2010 .

[15]  Steffen Lutz,et al.  77 GHz lens-based multistatic MIMO radar with colocated antennas for automotive applications , 2012, 2012 IEEE/MTT-S International Microwave Symposium Digest.

[16]  Carmine Clemente,et al.  Robust PCA micro-doppler classification using SVM on embedded systems , 2014, IEEE Transactions on Aerospace and Electronic Systems.

[17]  Matthias A. Hein,et al.  Monostatic RCS measurements of a passenger car mock-up at 77 GHz frequency in virtual environment , 2019, 2019 16th European Radar Conference (EuRAD).

[18]  H. Wechsler,et al.  Micro-Doppler effect in radar: phenomenon, model, and simulation study , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[19]  Daniel Thalmann,et al.  A global human walking model with real-time kinematic personification , 1990, The Visual Computer.

[20]  Dave Tahmoush,et al.  Radar Measurement of Human Polarimetric Micro-Doppler , 2013, J. Electr. Comput. Eng..

[21]  F. Groen,et al.  Human walking estimation with radar , 2003 .

[22]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[23]  Robert A. Kipp Curved surface scattering geometry in the shooting and bouncing rays method , 2010, 2010 IEEE Antennas and Propagation Society International Symposium.

[24]  Liting Zhou,et al.  Use of deep neural networks for classification of micro-Doppler signature from radar data , 2019 .

[25]  Fujitsu Ten Masayuki KISHIDA 79 GHz Band Ultra-Wideband Automotive Radar , 2014 .

[26]  Ying Liu,et al.  Recognition and Classification of Rotorcraft by Micro-Doppler Signatures Using Deep Learning , 2018, ICCS.

[27]  Alexander Ioffe,et al.  Micro-doppler based classifying features for automotive radar VRU target classification , 2017 .

[28]  Juan D. Castro,et al.  Enabling Safe Autonomous Vehicles by Advanced mm-Wave Radar Simulations , 2019, 2019 IEEE MTT-S International Microwave Symposium (IMS).

[29]  Wolfgang Menzel,et al.  Multi-target reflection point model of cyclists for automotive radar , 2017, 2017 European Radar Conference (EURAD).

[30]  Murat Torlak,et al.  Automotive Radars: A review of signal processing techniques , 2017, IEEE Signal Processing Magazine.

[31]  Andreas Koch,et al.  Micro-Doppler Extraction of Bicycle Pedaling Movements Using Automotive Radar , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[32]  Ushemadzoro Chipengo,et al.  A 77 GHz Simulation Study of Roadway Infrastructure Radar Signatures for Smart Roads , 2019, 2019 16th European Radar Conference (EuRAD).

[33]  Moeness G. Amin,et al.  DNN Transfer Learning From Diversified Micro-Doppler for Motion Classification , 2018, IEEE Transactions on Aerospace and Electronic Systems.

[34]  Urs Schneider,et al.  Real-time capable micro-Doppler signature decomposition of walking human limbs , 2017, 2017 IEEE Radar Conference (RadarConf).

[35]  Chengwei Zhou,et al.  FFT-Based DOA Estimation for Coprime MIMO Radar: A Hardware-Friendly Approach , 2018, 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP).

[36]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[37]  Xinhua Liu,et al.  Classification of Human Motions Using Micro-Doppler Radar in the Environments with Micro-Motion Interference , 2019, Sensors.

[38]  S. Z. Gürbüz,et al.  Deep learning of micro-Doppler features for aided and unaided gait recognition , 2017, 2017 IEEE Radar Conference (RadarConf).

[39]  Ralph Helmar Rasshofer,et al.  Pedestrian recognition using automotive radar sensors , 2012 .

[40]  Hao Ling,et al.  Simulation of human microDopplers using computer animation data , 2008, 2008 IEEE Radar Conference.

[41]  Hao Ling,et al.  Microdoppler signature simulation of computer animated human and animal motions , 2008, 2008 IEEE Antennas and Propagation Society International Symposium.

[42]  Shobha Sundar Ram,et al.  Database Of Simulated Inverse Synthetic Aperture Radar Images For Short Range Automotive Radar , 2020, 2020 IEEE International Radar Conference (RADAR).

[43]  Vehicle wheel detection using micro doppler effect , 2020 .

[44]  S. Julier,et al.  Augmenting Experimental Data with Simulations to Improve Activity Classification in Healthcare Monitoring , 2021, 2021 IEEE Radar Conference (RadarConf21).

[45]  Ashish Kumar Singh,et al.  Analysis of Human Kinetics Using Millimeter-Wave Micro-Doppler Radar , 2015, IHCI.

[46]  Ushemadzoro Chipengo,et al.  High Fidelity Physics Simulation of 128 Channel MIMO Sensor for 77GHz Automotive Radar , 2020, IEEE Access.

[47]  Samiur Rahman,et al.  Radar micro-Doppler signatures of drones and birds at K-band and W-band , 2018, Scientific Reports.

[48]  Rick S. Blum,et al.  MIMO radar: an idea whose time has come , 2004, Proceedings of the 2004 IEEE Radar Conference (IEEE Cat. No.04CH37509).

[49]  Ali Cafer Gürbüz,et al.  Knowledge Exploitation for Human Micro-Doppler Classification , 2015, IEEE Geoscience and Remote Sensing Letters.

[50]  Qiang Yi,et al.  Development of test scenarios and bicyclist surrogate for the evaluation of bicyclist automatic emergency braking systems , 2018 .

[51]  Ralph Helmar Rasshofer,et al.  Automotive Radar and Lidar Systems for Next Generation Driver Assistance Functions , 2005 .

[52]  Nojun Kwak,et al.  Human detection by Neural Networks using a low-cost short-range Doppler radar sensor , 2017, 2017 IEEE Radar Conference (RadarConf).

[53]  Shawn Carpenter,et al.  From Antenna Design to High Fidelity, Full Physics Automotive Radar Sensor Corner Case Simulation , 2018, Modelling and Simulation in Engineering.

[54]  Fredrik Gustafsson,et al.  Millimeter-wave radar micro-Doppler signatures of human motion , 2011, 2011 12th International Radar Symposium (IRS).

[55]  Arien P. Sligar,et al.  Machine Learning-Based Radar Perception for Autonomous Vehicles Using Full Physics Simulation , 2020, IEEE Access.

[56]  Seong-Cheol Kim,et al.  Human–vehicle classification using feature-based SVM in 77-GHz automotive FMCW radar , 2017 .

[57]  David Johnson,et al.  Radar Sensing for Intelligent Vehicles in Urban Environments , 2015, Sensors.

[58]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[59]  Sevgi Zubeyde Gurbuz,et al.  A kinect-based human micro-doppler simulator , 2015, IEEE Aerospace and Electronic Systems Magazine.

[60]  Youngwook Kim,et al.  Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[61]  Daegun Oh,et al.  Classification of Drone Type Using Deep Convolutional Neural Networks Based on Micro- Doppler Simulation , 2018, 2018 International Symposium on Antennas and Propagation (ISAP).

[62]  Shobha Sundar Ram,et al.  Simulation of the radar cross-section of dynamic human motions using virtual reality data and ray tracing , 2018, 2018 IEEE Radar Conference (RadarConf18).

[63]  Chi-Chih Chen,et al.  Bicycles and human riders backscattering at 77 GHz for automotive radar , 2016, 2016 10th European Conference on Antennas and Propagation (EuCAP).

[64]  Vinh Dang,et al.  Analysis of Moving Human Micro-Doppler Signature in Forest Environments , 2014 .