Gesture Recognition Using Ambient Light

There is a growing interest in the scientific community to develop techniques for humans to communicate with the computing that is embedding into our environments. Researchers are already exploring ubiquitous modalities, such as radio frequency signals, to develop gesture recognition systems. In this paper, we explore another such modality, namely ambient light, and develop LiGest, an ambient light based gesture recognition system. The key property of LiGest is that it is agnostic to lighting conditions, position and orientation of user, and who performs the gestures. The general idea behind LiGest is that when a user performs different gestures, the shadows of the user move in unique patterns. LiGest first learns these patterns using training samples and then recognizes unknown samples by matching them with the learnt patterns. To capture these patterns, LiGest uses a grid of light sensors deployed on floor. While the general idea behind LiGest seems straightforward, it is actually very challenging to put it into practice because the intensity, size, and number of shadows of a user are not fixed and depend highly on the position and orientation of a user as well as on the intensity, position, and number of light sources. We developed a prototype of LiGest using commercially available light sensors and extensively evaluated it with the help of 20 volunteers. Our results show that LiGest achieves an average accuracy of 96.36% across all volunteers.

[1]  Tianxing Li,et al.  Real-Time Screen-Camera Communication Behind Any Scene , 2015, MobiSys.

[2]  Inseok Hwang,et al.  E-Gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices , 2011, SenSys.

[3]  Shyamnath Gollakota,et al.  Feasibility and limits of wi-fi imaging , 2014, SenSys.

[4]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[5]  William T. Freeman,et al.  Bayesian Reconstruction of 3D Human Motion from Single-Camera Video , 1999, NIPS.

[6]  Paul Congdon,et al.  Avoiding multipath to revive inbuilding WiFi localization , 2013, MobiSys '13.

[7]  Zachary Weinberg,et al.  I Still Know What You Visited Last Summer: Leaking Browsing History via User Interaction and Side Channel Attacks , 2011, 2011 IEEE Symposium on Security and Privacy.

[8]  Zhihan Lv,et al.  Multimodal Hand and Foot Gesture Interaction for Handheld Devices , 2013, TOMM.

[9]  Guobin Shen,et al.  Epsilon: A Visible Light Based Positioning System , 2014, NSDI.

[10]  Yunhao Liu,et al.  Towards omnidirectional passive human detection , 2013, 2013 Proceedings IEEE INFOCOM.

[11]  Fadel Adib,et al.  Multi-Person Localization via RF Body Reflections , 2015, NSDI.

[12]  Yunhao Liu,et al.  From RSSI to CSI , 2013, ACM Comput. Surv..

[13]  B. Silverman,et al.  The Stationary Wavelet Transform and some Statistical Applications , 1995 .

[14]  A. Nelson,et al.  Wearable multi-sensor gesture recognition for paralysis patients , 2013, 2013 IEEE SENSORS.

[15]  Pascal Fua,et al.  Skeleton-based motion capture for robust reconstruction of human motion , 2000, Proceedings Computer Animation 2000.

[16]  Eric C. Larson,et al.  Dante vision: In-air and touch gesture sensing for natural surface interaction with combined depth and thermal cameras , 2012, 2012 IEEE International Conference on Emerging Signal Processing Applications.

[17]  Shyamnath Gollakota,et al.  Bringing Gesture Recognition to All Devices , 2014, NSDI.

[18]  Ashwin Ashok,et al.  Challenge: mobile optical networks through visual MIMO , 2010, MobiCom.

[19]  Khaled A. Harras,et al.  WiGest: A ubiquitous WiFi-based gesture recognition system , 2014, 2015 IEEE Conference on Computer Communications (INFOCOM).

[20]  J. Trumpf,et al.  Visible Spectrum Optical Communication and Distance Sensing for Underwater Applications , 2004 .

[21]  Steve Fotios,et al.  Lighting in offices: lamp spectrum and brightness , 2011 .

[22]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[23]  Muhammad Shahzad,et al.  Position and Orientation Agnostic Gesture Recognition Using WiFi , 2017, MobiSys.

[24]  Giles M. Foody,et al.  A relative evaluation of multiclass image classification by support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Prabal Dutta,et al.  Luxapose: indoor positioning with mobile phones and visible light , 2014, MobiCom.

[26]  Fadel Adib,et al.  Multi-Person Motion Tracking via RF Body Reflections , 2014 .

[27]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[28]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[29]  Jakub Segen,et al.  Shadow gestures: 3D hand pose estimation using a single camera , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[30]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[31]  Maarten Jansen,et al.  Noise Reduction by Wavelet Thresholding , 2001 .

[32]  Sang-Kook Han,et al.  An Indoor Visible Light Communication Positioning System Using a RF Carrier Allocation Technique , 2013, Journal of Lightwave Technology.

[33]  Euripides G. M. Petrakis,et al.  Motion sensor driven gesture recognition for future internet application development , 2014, IISA 2014, The 5th International Conference on Information, Intelligence, Systems and Applications.

[34]  Bernhard Schölkopf,et al.  Comparing support vector machines with Gaussian kernels to radial basis function classifiers , 1997, IEEE Trans. Signal Process..

[35]  Gregory D. Abowd,et al.  Automatic Synchronization of Wearable Sensors and Video-Cameras for Ground Truth Annotation -- A Practical Approach , 2012, 2012 16th International Symposium on Wearable Computers.

[36]  Edward W. Knightly,et al.  Enabling vehicular visible light communication (V2LC) networks , 2011, VANET '11.

[37]  Takeo Kanade,et al.  A real time system for robust 3D voxel reconstruction of human motions , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[38]  Shaojie Tang,et al.  Electronic frog eye: Counting crowd using WiFi , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[39]  Dina Katabi,et al.  PixNet: interference-free wireless links using LCD-camera pairs , 2010, MobiCom.

[40]  Vivek Sarkar,et al.  Automatic detection of inter-application permission leaks in Android applications , 2013, IBM J. Res. Dev..

[41]  Georg Umlauf,et al.  3D Hand Gesture Recognition Based on Sensor Fusion of Commodity Hardware , 2012, MuC.

[42]  HiltonAdrian,et al.  A survey of advances in vision-based human motion capture and analysis , 2006 .

[43]  Pamela J. Wisniewski,et al.  Multiwave: Doppler Effect Based Gesture Recognition in Multiple Dimensions , 2016, CHI Extended Abstracts.

[44]  Wei Wang,et al.  Understanding and Modeling of WiFi Signal Based Human Activity Recognition , 2015, MobiCom.

[45]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[46]  Xia Zhou,et al.  Human Sensing Using Visible Light Communication , 2015, MobiCom.

[47]  Desney S. Tan,et al.  SoundWave: using the doppler effect to sense gestures , 2012, CHI.

[48]  Darren Leigh,et al.  Very Low-Cost Sensing and Communication Using Bidirectional LEDs , 2003, UbiComp.

[49]  Serban Oprisescu,et al.  Automatic static hand gesture recognition using ToF cameras , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[50]  Mary Yamada,et al.  Adoption of Light-Emitting Diodes in Common Lighting Applications , 2013 .

[51]  Lorenzo Porzi,et al.  A smart watch-based gesture recognition system for assisting people with visual impairments , 2013, IMMPD '13.

[52]  Koji Yatani,et al.  BodyScope: a wearable acoustic sensor for activity recognition , 2012, UbiComp.

[53]  Stefan Schmid,et al.  LED-to-LED visible light communication networks , 2013, MobiHoc '13.

[54]  Robin R. Murphy,et al.  Hand gesture recognition with depth images: A review , 2012, 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.

[55]  Luc Van Gool,et al.  Real-time 3D hand gesture interaction with a robot for understanding directions from humans , 2011, 2011 RO-MAN.

[56]  Siddharth Shah,et al.  AutoSense: unobtrusively wearable sensor suite for inferring the onset, causality, and consequences of stress in the field , 2011, SenSys.

[57]  Yusheng Ji,et al.  RF-Sensing of Activities from Non-Cooperative Subjects in Device-Free Recognition Systems Using Ambient and Local Signals , 2014, IEEE Transactions on Mobile Computing.

[58]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[59]  Zhi Xu,et al.  TapLogger: inferring user inputs on smartphone touchscreens using on-board motion sensors , 2012, WISEC '12.

[60]  Diana Kaholokula Reusing Ambient Light to Recognize Hand Gestures Prepared by Mahina - , 2016 .

[61]  Yang Li,et al.  User-defined motion gestures for mobile interaction , 2011, CHI.

[62]  Jun Luo,et al.  CeilingSee: Device-free occupancy inference through lighting infrastructure based LED sensing , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[63]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2014, IEEE Transactions on Mobile Computing.

[64]  Peter A. Dinda,et al.  Sonar-based measurement of user presence and attention , 2009, UbiComp.

[65]  Yusheng Ji,et al.  Leveraging RF-channel fluctuation for activity recognition: Active and passive systems, continuous and RSSI-based signal features , 2013, MoMM '13.

[66]  Chunming Hu,et al.  Enhancing reliability to boost the throughput over screen-camera links , 2014, MobiCom.

[67]  Gerhard Tröster,et al.  The telepathic phone: Frictionless activity recognition from WiFi-RSSI , 2014, 2014 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[68]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[69]  Kaigui Bian,et al.  Strata: layered coding for scalable visual communication , 2014, MobiCom.

[70]  Yi Li,et al.  Hand gesture recognition using Kinect , 2012, 2012 IEEE International Conference on Computer Science and Automation Engineering.

[71]  Thomas D. C. Little,et al.  Using LED Lighting for Ubiquitous Indoor Wireless Networking , 2008, 2008 IEEE International Conference on Wireless and Mobile Computing, Networking and Communications.

[72]  Rafael Pérez Jiménez,et al.  Reading lamp-based visible light communication system for in-flight entertainment , 2013, IEEE Transactions on Consumer Electronics.

[73]  Guoliang Xing,et al.  COBRA: color barcode streaming for smartphone systems , 2012, MobiSys '12.

[74]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[75]  Toshiaki Fujii,et al.  Experimental on Hierarchical Transmission Scheme for Visible Light Communication using LED Traffic Light and High-Speed Camera , 2007, 2007 IEEE 66th Vehicular Technology Conference.

[76]  Kaishun Wu,et al.  WiFall: Device-free fall detection by wireless networks , 2017, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[77]  Michael W. Marcellin,et al.  An overview of JPEG-2000 , 2000, Proceedings DCC 2000. Data Compression Conference.

[78]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[79]  Qiang Liu,et al.  Practical Human Sensing in the Light , 2016, GETMBL.

[80]  Zhihan Lv,et al.  Foot motion sensing: augmented game interface based on foot interaction for smartphone , 2014, CHI Extended Abstracts.

[81]  Mark J. Shensa,et al.  The discrete wavelet transform: wedding the a trous and Mallat algorithms , 1992, IEEE Trans. Signal Process..

[82]  Sachin Katti,et al.  PinPoint: Localizing Interfering Radios , 2013, NSDI.

[83]  Chintan Patel,et al.  Inviz: Low-power personalized gesture recognition using wearable textile capacitive sensor arrays , 2015, PerCom.

[84]  Guijin Wang,et al.  A hand gesture based interactive presentation system utilizing heterogeneous cameras , 2012 .

[85]  M. Köppen,et al.  The Curse of Dimensionality , 2010 .

[86]  Weihua Sheng,et al.  Human gesture recognition through a Kinect sensor , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[87]  Wenjun Hu,et al.  LightSync: unsynchronized visual communication over screen-camera links , 2013, MobiCom.

[88]  Dawid Gradolewski,et al.  The Use of Wavelet Analysis to Denoising of Electrocardiography Signal. , 2013 .

[89]  Zyad Shaaban,et al.  Normalization as a Preprocessing Engine for Data Mining and the Approach of Preference Matrix , 2006, 2006 International Conference on Dependability of Computer Systems.

[90]  Pavlo Molchanov,et al.  Multi-sensor system for driver's hand-gesture recognition , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[91]  Mohsen Kavehrad,et al.  Comparison of VLC-based indoor positioning techniques , 2013, Photonics West - Optoelectronic Materials and Devices.

[92]  Zhihan Lv,et al.  Wearable Smartphone: Wearable Hybrid Framework for Hand and Foot Gesture Interaction on Smartphone , 2013, 2013 IEEE International Conference on Computer Vision Workshops.