We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatio-temporal properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.
[1]
Parth H. Pathak,et al.
mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals
,
2020,
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..
[2]
George Shaker,et al.
Glucose Levels Detection Using mm-Wave Radar
,
2018,
IEEE Sensors Letters.
[3]
George Shaker,et al.
Non-Invasive Monitoring of Glucose Level Changes Utilizing a mm-Wave Radar System
,
2018,
Int. J. Mob. Hum. Comput. Interact..
[4]
Shuangquan Wang,et al.
SignFi
,
2018,
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..
[5]
Wenbing Zhao,et al.
A Survey of Applications and Human Motion Recognition with Microsoft Kinect
,
2015,
Int. J. Pattern Recognit. Artif. Intell..
[6]
F.C. Robey,et al.
MIMO radar theory and experimental results
,
2004,
Conference Record of the Thirty-Eighth Asilomar Conference on Signals, Systems and Computers, 2004..
[7]
S. Rautaray,et al.
Vision based hand gesture recognition for human computer interaction: a survey
,
2012,
Artificial Intelligence Review.