SeleCon: Scalable IoT Device Selection and Control Using Hand Gestures

Although different interaction modalities have been proposed in the field of human-computer interface (HCI), only a few of these techniques could reach the end users because of scalability and usability issues. Given the popularity and the growing number of IoT devices, selecting one out of many devices becomes a hurdle in a typical smarthome environment. Therefore, an easy-to-learn, scalable, and non-intrusive interaction modality has to be explored. In this paper, we propose a pointing approach to interact with devices, as pointing is arguably a natural way for device selection. We introduce SeleCon for device selection and control which uses an ultra-wideband (UWB) equipped smartwatch. To interact with a device in our system, people can point to the device to select it then draw a hand gesture in the air to specify a control action. To this end, SeleCon employs inertial sensors for pointing gesture detection and a UWB transceiver for identifying the selected device from ranging measurements. Furthermore, SeleCon supports an alphabet of gestures that can be used for controlling the selected devices. We performed our experiment in a 9m-by-10m lab space with eight deployed devices. The results demonstrate that SeleCon can achieve 84.5% accuracy for device selection and 97% accuracy for hand gesture recognition. We also show that SeleCon is power efficient to sustain daily use by turning off the UWB transceiver, when a user's wrist is stationary.

[1]  Pan Jing,et al.  Human-computer Interaction using Pointing Gesture based on an Adaptive Virtual Touch Screen , 2013 .

[2]  Kaishun Wu,et al.  WiFall: Device-Free Fall Detection by Wireless Networks , 2017, IEEE Transactions on Mobile Computing.

[3]  Joseph A. Paradiso,et al.  WristQue: A personal sensor wristband , 2013, 2013 IEEE International Conference on Body Sensor Networks.

[4]  Paul Lukowicz,et al.  Continuous recognition of arm activities with body-worn inertial sensors , 2004, Eighth International Symposium on Wearable Computers.

[5]  Ming-Syan Chen,et al.  HeatProbe: a thermal-based power meter for accounting disaggregated electricity usage , 2011, UbiComp '11.

[6]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2014, IEEE Transactions on Mobile Computing.

[7]  Xiang-Yang Li,et al.  You're driving and texting: detecting drivers using personal smart phones by leveraging inertial sensors , 2013, MobiCom.

[8]  Shyamnath Gollakota,et al.  Wi-Fi Gesture Recognition on Existing Devices , 2014, ArXiv.

[9]  Kang Ryoung Park,et al.  Face Recognition System for Set-Top Box-Based Intelligent TV , 2014, Sensors.

[10]  Shaojie Tang,et al.  Electronic frog eye: Counting crowd using WiFi , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[11]  Jun Han,et al.  ACComplice: Location inference using accelerometers on smartphones , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[12]  Eric C. Larson,et al.  HeatWave: thermal imaging for surface user interaction , 2011, CHI.

[13]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[14]  Desney S. Tan,et al.  SoundWave: using the doppler effect to sense gestures , 2012, CHI.

[15]  Desney S. Tan,et al.  Humantenna: using the body as an antenna for real-time whole-body interaction , 2012, CHI.

[16]  G.B. Giannakis,et al.  Localization via ultra-wideband radios: a look at positioning aspects for future sensor networks , 2005, IEEE Signal Processing Magazine.

[17]  Romit Roy Choudhury,et al.  Using mobile phones to write in air , 2011, MobiSys '11.

[18]  Fernando Alonso-Martín,et al.  Augmented Robotics Dialog System for Enhancing Human–Robot Interaction , 2015, Sensors.

[19]  ZhouZimu,et al.  We Can Hear You with Wi-Fi! , 2016 .

[20]  Fadel Adib,et al.  See through walls with WiFi! , 2013, SIGCOMM.

[21]  Kongqiao Wang,et al.  Hand gesture recognition and virtual game control based on 3D accelerometer and EMG sensors , 2009, IUI.

[22]  Vahid Tarokh,et al.  UWB indoor path loss model for residential and commercial buildings , 2003, 2003 IEEE 58th Vehicular Technology Conference. VTC 2003-Fall (IEEE Cat. No.03CH37484).

[23]  Lars Kulik,et al.  Gesture recognition using RFID technology , 2012, Personal and Ubiquitous Computing.

[24]  Begoña García Zapirain,et al.  Eye/Head Tracking Technology to Improve HCI with iPad Applications , 2015, Sensors.

[25]  He Wang,et al.  I am a Smartwatch and I can Track my User's Arm , 2016, MobiSys.

[26]  Wei Wang,et al.  Keystroke Recognition Using WiFi Signals , 2015, MobiCom.

[27]  Takeo Kanade,et al.  Visual Tracking of High DOF Articulated Structures: an Application to Human Hand Tracking , 1994, ECCV.

[28]  Patrick Traynor,et al.  (sp)iPhone: decoding vibrations from nearby keyboards using mobile phone accelerometers , 2011, CCS '11.

[29]  Chung-Lin Huang,et al.  Hand gesture recognition using a real-time tracking method and hidden Markov models , 2003, Image Vis. Comput..

[30]  Babak Naderi,et al.  Magnetic signatures in air for mobile devices , 2012, Mobile HCI.

[31]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[32]  Saïda Bouakaz,et al.  Real-Time and Markerless 3D Human Motion Capture Using Multiple Views , 2007, Workshop on Human Motion.

[33]  Yang Xu,et al.  WiFinger: talk to your smart devices with finger-grained gesture , 2016, UbiComp.

[34]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[35]  Youngwook Kim,et al.  Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Anind K. Dey,et al.  Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch , 2016, CHI.

[37]  Jun Han,et al.  ACCessory: password inference using accelerometers on smartphones , 2012, HotMobile '12.

[38]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[39]  Inseok Hwang,et al.  E-Gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices , 2011, SenSys.

[40]  Roger Wattenhofer,et al.  Recognizing text using motion data from a smartwatch , 2016, 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[41]  Parth H. Pathak,et al.  Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch , 2015, HotMobile.

[42]  Lorenzo Porzi,et al.  A smart watch-based gesture recognition system for assisting people with visual impairments , 2013, IMMPD '13.

[43]  Kevin Bouchard,et al.  Gesture recognition in smart home using passive RFID technology , 2014, PETRA.

[44]  Jun Rekimoto,et al.  GestureWrist and GesturePad: unobtrusive wearable interaction devices , 2001, Proceedings Fifth International Symposium on Wearable Computers.