Device free human gesture recognition using Wi-Fi CSI: A survey

Abstract Device-free sensing of human gestures has gained tremendous research attention with the recent advancements in wireless technologies. Channel State Information (CSI), a metric of Wi-Fi devices adopted for device-free sensing achieves better recognition performance. This survey classifies the state of the art recognition task into device-based and device-free sensing methods and highlights advancements with Wi-Fi CSI. This paper also comprehensively summarizes the recognition performance of device-free sensing using CSI under two approaches: model-based and learning based approaches. Machine Learning and Deep Learning algorithms are discussed under the learning based approaches with its corresponding recognition accuracy. Various signal pre-processing, feature extraction, selection, and classification techniques that are widely adopted for gesture recognition along with the environmental factors that influence the recognition accuracy are also discussed. This survey presents the conclusion spotting the challenges and opportunities that could be explored in the device free gesture recognition using the CSI metric of Wi-Fi devices.

[1]  Yingxu Wang,et al.  Kinect Sensor Gesture and Activity Recognition: New Applications for Consumer Cognitive Systems , 2018, IEEE Consumer Electronics Magazine.

[2]  Hao Jiang,et al.  Device-Free Occupant Activity Sensing Using WiFi-Enabled IoT Devices for Smart Homes , 2018, IEEE Internet of Things Journal.

[3]  Lei Shu,et al.  Locating in Crowdsourcing-Based DataSpace: Wireless Indoor Localization without Special Devices , 2014, Mob. Networks Appl..

[4]  Sung Ho Cho,et al.  Finger-Counting-Based Gesture Recognition within Cars Using Impulse Radar with Convolutional Neural Network , 2019, Sensors.

[5]  Xiaojiang Du,et al.  Robust WLAN-Based Indoor Intrusion Detection Using PHY Layer Information , 2018, IEEE Access.

[6]  Xiang Wang,et al.  WFID: Passive Device-free Human Identification Using WiFi Signal , 2016, MobiQuitous.

[7]  Jie Wu,et al.  Fine-grained vital signs estimation using commercial wi-fi devices , 2016, S3@MobiCom.

[8]  Ahmad Zaki Shukor,et al.  A New Data Glove Approach for Malaysian Sign Language Detection , 2015 .

[9]  Ian Howard,et al.  Localization of Bluetooth Smart Equipped Assets Based on Building Information Models , 2019 .

[10]  Muhammad Shahzad,et al.  Position and Orientation Agnostic Gesture Recognition Using WiFi , 2017, MobiSys.

[11]  Qian Zhang,et al.  Privacy-Preserving Location Authentication in Wi-Fi Networks Using Fine-Grained Physical Layer Signatures , 2016, IEEE Transactions on Wireless Communications.

[12]  Christopher W. Badenhop,et al.  The Z-Wave routing protocol and its security implications , 2017, Comput. Secur..

[13]  Peter E.D. Love,et al.  A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory , 2018 .

[14]  Swee King Phang,et al.  Higher Order Feature Extraction and Selection for Robust Human Gesture Recognition using CSI of COTS Wi-Fi Devices , 2019, Sensors.

[15]  S. Khawandi,et al.  Integrated monitoring system for fall detection in elderly , 2012, 2012 International Conference on Audio, Language and Image Processing.

[16]  Thomas Janssen,et al.  Outdoor Fingerprinting Localization Using Sigfox , 2018, 2018 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[17]  Jin Zhang,et al.  WiFi-ID: Human Identification Using WiFi Signal , 2016, 2016 International Conference on Distributed Computing in Sensor Systems (DCOSS).

[18]  Kay Römer,et al.  An Experimental Evaluation of the Reliability of LoRa Long-Range Low-Power Wireless Communication , 2017, J. Sens. Actuator Networks.

[19]  Fangmin Li,et al.  WiGeR: WiFi-Based Gesture Recognition System , 2016, ISPRS Int. J. Geo Inf..

[20]  Dirk Pesch,et al.  Channel State Information Based Human Presence Detection using Non-linear Techniques , 2016, BuildSys@SenSys.

[21]  Panlong Yang,et al.  R-TTWD: Robust Device-Free Through-The-Wall Detection of Moving Human With WiFi , 2017, IEEE Journal on Selected Areas in Communications.

[22]  Yusheng Ji,et al.  MoSense: An RF-Based Motion Detection System via Off-the-Shelf WiFi Devices , 2017, IEEE Internet of Things Journal.

[23]  Ümit Y. Ogras,et al.  Online Human Activity Recognition using Low-Power Wearable Devices , 2018, 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[24]  Wei Wang,et al.  Device-Free Human Activity Recognition Using Commercial WiFi Devices , 2017, IEEE Journal on Selected Areas in Communications.

[25]  Jun-ichi Shirakashi,et al.  Gesture Prediction Using Wearable Sensing Systems with Neural Networks for Temporal Data Analysis , 2019, Sensors.

[26]  Henry A. Kautz,et al.  Location-Based Activity Recognition using Relational Markov Networks , 2005, IJCAI.

[27]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[28]  L. M. Kamarudin,et al.  Analysis of RSSI-based DFL for human detection in indoor environment using IRIS mote , 2016, 2016 3rd International Conference on Electronic Design (ICED).

[29]  Dan Wu,et al.  WiDir: walking direction estimation using wireless signals , 2016, UbiComp.

[30]  Wei Zhou,et al.  A research on CSI-based human motion detection in complex scenarios , 2017, 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom).

[31]  Xiaolong Yang,et al.  WiCatch: A Wi-Fi Based Hand Gesture Recognition System , 2018, IEEE Access.

[32]  Li Sun,et al.  WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices , 2015, MobiCom.

[33]  Yi Wang,et al.  Robust Indoor Human Activity Recognition Using Wireless Signals , 2015, Sensors.

[34]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[35]  Van-Su Tran,et al.  A multi-storey building actuator and sensor system using 6L0WPAN based Internet of Things: Practical design and implementation , 2018, 2018 2nd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom).

[36]  Kaishun Wu,et al.  WiFall: Device-free fall detection by wireless networks , 2017, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[37]  Sandeep Rao,et al.  Real-Time Multi-Gesture Recognition using 77 GHz FMCW MIMO Single Chip Radar , 2019, 2019 IEEE International Conference on Consumer Electronics (ICCE).

[38]  Youngwook Kim,et al.  Application of Doppler radar for the recognition of hand gestures using optimized deep convolutional neural networks , 2017, 2017 11th European Conference on Antennas and Propagation (EUCAP).

[39]  Michael Weber,et al.  Find my stuff: supporting physical objects search with relative positioning , 2013, UbiComp.

[40]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2016, IEEE Trans. Mob. Comput..

[41]  Sriparna Saha,et al.  Gesture Recognition from Two-Person Interactions Using Ensemble Decision Tree , 2018 .

[42]  Fadel Adib,et al.  Emotion recognition using wireless signals , 2018, Commun. ACM.

[43]  Daniel Borrajo,et al.  A dynamic sliding window approach for activity recognition , 2011, UMAP'11.

[44]  Robin J. Evans,et al.  Hand-Gesture Recognition Using Two-Antenna Doppler Radar With Deep Convolutional Neural Networks , 2019, IEEE Sensors Journal.

[45]  Weijia Jia,et al.  WiFind: Driver Fatigue Detection with Fine-Grained Wi-Fi Signal Features , 2020, IEEE Transactions on Big Data.

[46]  Arjan Kuijper,et al.  Fitness Activity Recognition on Smartphones Using Doppler Measurements , 2018, Informatics.

[47]  Linqing Gui,et al.  SEARE: A System for Exercise Activity Recognition and Quality Evaluation Based on Green Sensing , 2018, IEEE Transactions on Emerging Topics in Computing.

[48]  Mo Li,et al.  Precise Power Delay Profiling with Commodity Wi-Fi , 2019, IEEE Trans. Mob. Comput..

[49]  Meng-Luen Wu,et al.  Real-Time Musical Conducting Gesture Recognition Based on a Dynamic Time Warping Classifier Using a Single-Depth Camera , 2019, Applied Sciences.

[50]  Khaled A. Harras,et al.  WiGest: A ubiquitous WiFi-based gesture recognition system , 2014, 2015 IEEE Conference on Computer Communications (INFOCOM).

[51]  Jijun Zhao,et al.  R-DEHM: CSI-Based Robust Duration Estimation of Human Motion with WiFi , 2019, Sensors.

[52]  Muhammad Shahzad,et al.  Multi-User Gesture Recognition Using WiFi , 2018, MobiSys.

[53]  Minyi Guo,et al.  Learning Human Activities through Wi-Fi Channel State Information with Multiple Access Points , 2018, IEEE Communications Magazine.

[54]  Wei Wang,et al.  Gait recognition using wifi signals , 2016, UbiComp.

[55]  Juan-Carlos Cano,et al.  Crowdsensing in Smart Cities: Overview, Platforms, and Environment Sensing Issues , 2018, Sensors.

[56]  Miao Yu,et al.  WiFi-Based Real-Time Calibration-Free Passive Human Motion Detection † , 2015, Sensors.

[57]  Qian Lei,et al.  The application of the Narrowband-IoT in the smoke alarm , 2018 .

[58]  Kate Ching-Ju Lin,et al.  WiFi action recognition via vision-based methods , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[59]  Yusheng Ji,et al.  RF-Sensing of Activities from Non-Cooperative Subjects in Device-Free Recognition Systems Using Ambient and Local Signals , 2014, IEEE Transactions on Mobile Computing.

[60]  Xing Zhou,et al.  WiStep , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[61]  D. Girbau,et al.  Near-Field Soil Moisture Sensor with Energy Harvesting Capability , 2018, 2018 48th European Microwave Conference (EuMC).

[62]  Jie He,et al.  WiDriver: Driver Activity Recognition System Based on WiFi CSI , 2018, Int. J. Wirel. Inf. Networks.

[63]  Yang Liu,et al.  Accurate Device-Free Tracking Using Inexpensive RFIDs , 2018, Sensors.

[64]  Dan Wu,et al.  Toward Centimeter-Scale Human Activity Sensing with Wi-Fi Signals , 2017, Computer.

[65]  Akira Fukuda,et al.  Danger-Pose Detection System Using Commodity Wi-Fi for Bathroom Monitoring , 2019, Sensors.

[66]  Parth H. Pathak,et al.  Analyzing Shopper's Behavior through WiFi Signals , 2015, WPA@MobiSys.

[67]  Mi Zhang,et al.  HeadScan: A Wearable System for Radio-Based Sensing of Head and Mouth-Related Activities , 2016, 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[68]  Bernt Schiele,et al.  ADL recognition based on the combination of RFID and accelerometer sensing , 2008, Pervasive 2008.

[69]  Yunhao Liu,et al.  Enabling Contactless Detection of Moving Humans with Dynamic Speeds Using CSI , 2018, ACM Trans. Embed. Comput. Syst..

[70]  Xiang Zhang,et al.  WiHumidity: A Novel CSI-Based Humidity Measurement System , 2016, SmartCom.

[71]  Zhu Wang,et al.  FreeSense: human-behavior understanding using Wi-Fi signals , 2018, J. Ambient Intell. Humaniz. Comput..

[72]  Jie Wang,et al.  CSI-Based Device-Free Wireless Localization and Activity Recognition Using Radio Image Features , 2017, IEEE Transactions on Vehicular Technology.

[73]  Shaojie Tang,et al.  Wi-Sleep: Contactless Sleep Monitoring via WiFi Signals , 2014, 2014 IEEE Real-Time Systems Symposium.

[74]  Yusheng Ji,et al.  Passive detection of situations from ambient FM-radio signals , 2012, UbiComp.

[75]  Xiang Li,et al.  AR-Alarm: An Adaptive and Robust Intrusion Detection System Leveraging CSI from Commodity Wi-Fi , 2017, ICOST.

[76]  Julián Colorado,et al.  IoT system for Human Activity Recognition using BioHarness 3 and Smartphone , 2017, ICFNDS.

[77]  Xiaohui Xie,et al.  Invisible Cloak Fails: CSI-based Passive Human Detection , 2015, CSAR@SenSys.

[78]  Jakub Galka,et al.  Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition , 2016, IEEE Sensors Journal.

[79]  Jian Lu,et al.  epSICAR: An Emerging Patterns based approach to sequential, interleaved and Concurrent Activity Recognition , 2009, 2009 IEEE International Conference on Pervasive Computing and Communications.

[80]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[81]  Rui Zhou,et al.  Device-Free Presence Detection and Localization With SVM and CSI Fingerprinting , 2017, IEEE Sensors Journal.

[82]  Bing Chen,et al.  WiCount: A Deep Learning Approach for Crowd Counting Using WiFi Signals , 2017, 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC).

[83]  Yongdae Kim,et al.  GUTI Reallocation Demystified: Cellular Location Tracking with Changing Temporary Identifier , 2018, NDSS.

[84]  Yusuke Doi,et al.  Study of LoRaWAN Technology for Activity Recognition , 2018, UbiComp/ISWC Adjunct.

[85]  Marina Ruggieri,et al.  WIBECAM: Device Free Human Activity Recognition Through WiFi Beacon-Enabled Camera , 2015, WPA@MobiSys.

[86]  Wang Lin,et al.  From Lens to Prism: Device-Free Modeling and Recognition of Multi-Part Activities , 2018, IEEE Access.

[87]  Kannan Srinivasan,et al.  Mudra: User-friendly Fine-grained Gesture Recognition using WiFi Signals , 2016, CoNEXT.

[88]  Wei Wang,et al.  Keystroke Recognition Using WiFi Signals , 2015, MobiCom.

[89]  Muhammad Mahtab Alam,et al.  NarrowBand-IoT Performance Analysis for Healthcare Applications , 2018, ANT/SEIT.

[90]  Sangyoun Lee,et al.  Depth Camera-Based 3D Hand Gesture Controls with Immersive Tactile Feedback for Natural Mid-Air Gesture Interactions , 2015, Sensors.

[91]  Wei Gong,et al.  Channel Selective Activity Recognition with WiFi: A Deep Learning Approach Exploring Wideband Information , 2020, IEEE Transactions on Network Science and Engineering.

[92]  Hyunsoo Lee,et al.  The Effects of Housing Environments on the Performance of Activity-Recognition Systems Using Wi-Fi Channel State Information: An Exploratory Study , 2019, Sensors.

[93]  Weijia Jia,et al.  WiFind: Driver Fatigue Detection with Fine-Grained Wi-Fi Signal Features , 2017, GLOBECOM 2017 - 2017 IEEE Global Communications Conference.

[94]  K. J. Ray Liu,et al.  WiSpeed: A Statistical Electromagnetic Approach for Device-Free Indoor Speed Estimation , 2017, IEEE Internet of Things Journal.

[95]  Javier Del Ser,et al.  A Deep Learning Approach to Device-Free People Counting from WiFi Signals , 2018, IDC.

[96]  Yunhao Liu,et al.  Non-Invasive Detection of Moving and Stationary Human With WiFi , 2015, IEEE Journal on Selected Areas in Communications.

[97]  Zhe Chen,et al.  Syncope Detection in Toilet Environments Using Wi-Fi Channel State Information , 2018, UbiComp/ISWC Adjunct.

[98]  Yunhao Liu,et al.  Widar: Decimeter-Level Passive Tracking via Velocity Monitoring with Commodity Wi-Fi , 2017, MobiHoc.

[99]  Masatoshi Sekine,et al.  Activity Recognition Using Radio Doppler Effect for Human Monitoring Service , 2012, J. Inf. Process..

[100]  Xiaochao Dang,et al.  A Novel Passive Indoor Localization Method by Fusion CSI Amplitude and Phase Information , 2019, Sensors.

[101]  Shaojie Tang,et al.  Contactless Respiration Monitoring Via Off-the-Shelf WiFi Devices , 2016, IEEE Transactions on Mobile Computing.

[102]  Shyamnath Gollakota,et al.  Bringing Gesture Recognition to All Devices , 2014, NSDI.

[103]  Daqing Zhang,et al.  Anti-fall: A Non-intrusive and Real-Time Fall Detector Leveraging CSI from Commodity WiFi Devices , 2015, ICOST.

[104]  Yiming Pi,et al.  Dynamic Gesture Recognition with a Terahertz Radar Based on Range Profile Sequences and Doppler Signatures , 2017, Sensors.

[105]  Debi Prosad Dogra,et al.  A novel framework of continuous human-activity recognition using Kinect , 2018, Neurocomputing.

[106]  Kamin Whitehouse,et al.  Peripheral WiFi Vision: Exploiting Multipath Reflections for More Sensitive Human Sensing , 2017, WPA@MobiSys.

[107]  Tahsina Farah Sanam,et al.  FuseLoc: A CCA Based Information Fusion for Indoor Localization Using CSI Phase and Amplitude of Wifi Signals , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[108]  Khelifa Benahmed,et al.  Security Mechanisms for 6LoWPAN Network in Context of Internet of Things: A Survey , 2018, Renewable Energy for Smart and Sustainable Cities.

[109]  Sheng Tan,et al.  WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition , 2016, MobiHoc.

[110]  Han Zou,et al.  Towards occupant activity driven smart buildings via WiFi-enabled IoT devices and deep learning , 2018, Energy and Buildings.

[111]  Mi Zhang,et al.  BodyScan: Enabling Radio-based Sensing on Wearable Devices for Contactless Activity and Vital Sign Monitoring , 2016, MobiSys.

[112]  Haibin Cai,et al.  An adaptive wireless passive human detection via fine-grained physical layer information , 2016, Ad Hoc Networks.