A Novel Gesture Recognition System Based on CSI Extracted from a Smartphone with Nexmon Firmware

In this paper, we propose a novel gesture recognition system based on a smartphone. Due to the limitation of Channel State Information (CSI) extraction equipment, existing WiFi-based gesture recognition is limited to the microcomputer terminal equipped with Intel 5300 or Atheros 9580 network cards. Therefore, accurate gesture recognition can only be performed in an area relatively fixed to the transceiver link. The new gesture recognition system proposed by us breaks this limitation. First, we use nexmon firmware to obtain 256 CSI subcarriers from the bottom layer of the smartphone in IEEE 802.11ac mode on 80 MHz bandwidth to realize the gesture recognition system’s mobility. Second, we adopt the cross-correlation method to integrate the extracted CSI features in the time and frequency domain to reduce the influence of changes in the smartphone location. Third, we use a new improved DTW algorithm to classify and recognize gestures. We implemented vast experiments to verify the system’s recognition accuracy at different distances in different directions and environments. The results show that the system can effectively improve the recognition accuracy.

[1]  Fadel Adib,et al.  See through walls with WiFi! , 2013, SIGCOMM.

[2]  Youngwook Kim,et al.  Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[3]  Muhammad Shahzad,et al.  Position and Orientation Agnostic Gesture Recognition Using WiFi , 2017, MobiSys.

[4]  Fadel Adib,et al.  Emotion recognition using wireless signals , 2016, MobiCom.

[5]  Tommi S. Jaakkola,et al.  Learning Sleep Stages from Radio Signals: A Conditional Adversarial Architecture , 2017, ICML.

[6]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[7]  Wei Wang,et al.  Device-Free Human Activity Recognition Using Commercial WiFi Devices , 2017, IEEE Journal on Selected Areas in Communications.

[8]  Ting-Han Chen,et al.  WiTrack: Human-to-Human Mobility Relationship Tracking in Indoor Environments Based on Spatio-Temporal Wireless Signal Strength , 2019, 2019 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech).

[9]  Matthias Hollick,et al.  Shadow Wi-Fi: Teaching Smartphones to Transmit Raw Signals and to Extract Channel State Information to Implement Practical Covert Channels over Wi-Fi , 2018, MobiSys.

[10]  Chenglin Miao,et al.  Towards Environment Independent Device Free Human Activity Recognition , 2018, MobiCom.

[11]  Suresh Venkatasubramanian,et al.  Multiple Target Tracking with RF Sensor Networks , 2013, IEEE Transactions on Mobile Computing.

[12]  Kaishun Wu,et al.  WiFall: Device-free fall detection by wireless networks , 2017, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[13]  Fadel Adib,et al.  Multi-Person Localization via RF Body Reflections , 2015, NSDI.

[14]  Zheng Wang,et al.  CrossSense: Towards Cross-Site and Large-Scale WiFi Sensing , 2018, MobiCom.

[15]  Frédo Durand,et al.  Capturing the human figure through a wall , 2015, ACM Trans. Graph..

[16]  Sheng Tan,et al.  WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition , 2016, MobiHoc.

[17]  Mo Li,et al.  Precise Power Delay Profiling with Commodity Wi-Fi , 2015, IEEE Transactions on Mobile Computing.

[18]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[19]  Mi Zhang,et al.  HeadScan: A Wearable System for Radio-Based Sensing of Head and Mouth-Related Activities , 2016, 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[20]  Matthias Hollick,et al.  Free Your CSI: A Channel State Information Extraction Platform For Modern Wi-Fi Chipsets , 2019, WiNTECH.

[21]  Xiang Li,et al.  IndoTrack , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[22]  Chen Wang,et al.  Low Human-Effort, Device-Free Localization with Fine-Grained Subcarrier Information , 2018, IEEE Transactions on Mobile Computing.

[23]  Qiang Liu,et al.  Practical Human Sensing in the Light , 2016, GETMBL.

[24]  He Wang,et al.  I am a Smartwatch and I can Track my User's Arm , 2016, MobiSys.

[25]  Muhammad Shahzad,et al.  Augmenting User Identification with WiFi Based Gesture Recognition , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[26]  Khaled A. Harras,et al.  UbiBreathe: A Ubiquitous non-Invasive WiFi-based Breathing Estimator , 2015, MobiHoc.

[27]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[28]  Mi Zhang,et al.  BodyScan: Enabling Radio-based Sensing on Wearable Devices for Contactless Activity and Vital Sign Monitoring , 2016, MobiSys.

[29]  David Wetherall,et al.  Tool release: gathering 802.11n traces with channel state information , 2011, CCRV.

[30]  Muhammad Shahzad,et al.  Multi-User Gesture Recognition Using WiFi , 2018, MobiSys.

[31]  Li Sun,et al.  WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices , 2015, MobiCom.

[32]  Sachin Katti,et al.  WiDeo: Fine-grained Device-free Motion Tracing using RF Backscatter , 2015, NSDI.

[33]  Kaiming He,et al.  Detecting and Recognizing Human-Object Interactions , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Yunhao Liu,et al.  Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi , 2019, MobiSys.

[35]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[36]  Bingbing Ni,et al.  Recurrent Modeling of Interaction Context for Collective Activity Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[38]  Wei Wang,et al.  Keystroke Recognition Using WiFi Signals , 2015, MobiCom.

[39]  Shaojie Tang,et al.  Wi-Sleep: Contactless Sleep Monitoring via WiFi Signals , 2014, 2014 IEEE Real-Time Systems Symposium.

[40]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[41]  Xiang Li,et al.  Dynamic-MUSIC: accurate device-free indoor localization , 2016, UbiComp.

[42]  Nan Yu,et al.  QGesture , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[43]  Khaled A. Harras,et al.  WiGest: A ubiquitous WiFi-based gesture recognition system , 2014, 2015 IEEE Conference on Computer Communications (INFOCOM).

[44]  Eric C. Larson,et al.  DopLink: using the doppler effect for multi-device interaction , 2013, UbiComp.

[45]  Yunhao Liu,et al.  Widar: Decimeter-Level Passive Tracking via Velocity Monitoring with Commodity Wi-Fi , 2017, MobiHoc.

[46]  Kaishun Wu,et al.  WiFall: Device-Free Fall Detection by Wireless Networks , 2017, IEEE Transactions on Mobile Computing.