Mudra: User-friendly Fine-grained Gesture Recognition using WiFi Signals

There has been a great interest in recognizing gestures using wireless communication signals. We are motivated in detecting extremely fine, subtle finger gestures with WiFi signals. We envision this technology to find applications in finger-gesture control, disabled-friendly devices, physical therapy etc. The requirements of mm-level sensitivity and user-friendly feature using existing WiFi signals pose great challenges. Here, we present Mudra, a fine-grained finger gesture recognition system which leverages WiFi signals to enable a near-human-to-machine interaction with finger motion. Mudra uses a two-antenna receiver to detect and recognize finger gesture. It uses the signals received from one antenna to cancel the signal from the other. This "cancellation" is extremely sensitive to and enables us detect small variation in channel due to finger movements. Since Mudra decodes gestures with existing WiFi transmissions, Mudra enables gesture recognition without sacrificing WiFi transmission opportunities. Besides, Mudra is user-friendly with no need of user training. To demonstrate Mudra, we implement prototype on the NI-based SDR platform and use COTS WiFi adapter. We evaluate Mudra in a typical office environment. The results show that our system can achieve 96% accuracy.

[1]  Fadel Adib,et al.  See through walls with WiFi! , 2013, SIGCOMM.

[2]  Jörg Widmer,et al.  Survey on Energy Consumption Entities on the Smartphone Platform , 2011, 2011 IEEE 73rd Vehicular Technology Conference (VTC Spring).

[3]  Wei Wang,et al.  Keystroke Recognition Using WiFi Signals , 2015, MobiCom.

[4]  D. Katabi,et al.  JMB: scaling wireless capacity with user demands , 2012, CCRV.

[5]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2014, IEEE Transactions on Mobile Computing.

[6]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[7]  Yang Xiao,et al.  IEEE 802.11n: enhancements for higher throughput in wireless LANs , 2005, IEEE Wirel. Commun..

[8]  Shyamnath Gollakota,et al.  Bringing Gesture Recognition to All Devices , 2014, NSDI.

[9]  Bo Chen,et al.  Tracking Keystrokes Using Wireless Signals , 2015, MobiSys.

[10]  Desney S. Tan,et al.  FingerIO: Using Active Sonar for Fine-Grained Finger Tracking , 2016, CHI.

[11]  Ashutosh Sabharwal,et al.  Passive Self-Interference Suppression for Full-Duplex Infrastructure Nodes , 2013, IEEE Transactions on Wireless Communications.

[12]  Gaetano Borriello,et al.  Mobile Touch-Free Interaction for Global Health , 2015, HotMobile.

[13]  Martin Vuagnoux,et al.  Compromising Electromagnetic Emanations of Wired and Wireless Keyboards , 2009, USENIX Security Symposium.

[14]  Yang Xiao,et al.  IEEE 802.11n: enhancements for higher throughput in wireless LANs , 2005, IEEE Wireless Communications.

[15]  Dina Katabi,et al.  Zigzag decoding: combating hidden terminals in wireless networks , 2008, SIGCOMM '08.

[16]  Sachin Katti,et al.  WiDeo: Fine-grained Device-free Motion Tracing using RF Backscatter , 2015, NSDI.

[17]  Lei Yang,et al.  Tagoram: real-time tracking of mobile RFID tags to high precision using COTS devices , 2014, MobiCom.

[18]  Sangki Yun,et al.  Turning a Mobile Device into a Mouse in the Air , 2015, MobiSys.

[19]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[20]  Karl Woodbridge,et al.  A real-time high resolution passive WiFi Doppler-radar and its applications , 2014, 2014 International Radar Conference.

[21]  Xinyu Zhang,et al.  Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization , 2014, MobiSys.

[22]  Shyamnath Gollakota,et al.  Contactless Sleep Apnea Detection on Smartphones , 2015, GetMobile Mob. Comput. Commun..

[23]  Bo Chen,et al.  AirExpress: Enabling Seamless In-band Wireless Multi-hop Transmission , 2015, MobiCom.

[24]  Rob Miller,et al.  Smart Homes that Monitor Breathing and Heart Rate , 2015, CHI.