Leveraging Earables for Natural Calibration-Free Multi-Device Identification in Smart Environments

We present Look&Lock: a novel ubiquitous system that utilizes BLE communication between smart appliances in the environment and earables worn by the user to track head orientation and determine which device she is looking at. By harnessing this information, it is possible to lock on that device in a multi-device environment and allow only it to respond to user gestures without ambiguity. Our system leverages commercial off-the-shelf earables to provide accurate training-free head orientation tracking that is robust in different room settings. We implement a prototype using Android phones and eSense, a commercially available multi-sensory personal earable device. Our evaluation shows that Look&Lock can correctly identify devices inside a user's field of view with accuracy up to 100% and is robust to different configurations and room settings.

[1]  Michael Philippsen,et al.  Head-to-Body-Pose Classification in No-Pose VR Tracking Systems , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[2]  Dima Damen,et al.  Estimating visual attention from a head mounted IMU , 2015, SEMWEB.

[3]  Khaled Harras,et al.  Hitting Three Birds with One System: A Voice-Based CAPTCHA for the Modern User , 2018, 2018 IEEE International Conference on Web Services (ICWS).

[4]  Zhen Qin,et al.  Social Grouping for Multi-Target Tracking and Head Pose Estimation in Video , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Khaled A. Harras,et al.  EdgeHealth: An Energy-Efficient Edge-based Remote mHealth Monitoring System , 2019, 2019 IEEE Wireless Communications and Networking Conference (WCNC).

[6]  Khaled A. Harras,et al.  A Ubiquitous WiFi-Based Fine-Grained Gesture Recognition System , 2019, IEEE Transactions on Mobile Computing.

[7]  Khaled A. Harras,et al.  UbiBreathe: A Ubiquitous non-Invasive WiFi-based Breathing Estimator , 2015, MobiHoc.

[8]  Norihiro Hagita,et al.  Speech activity detection and face orientation estimation using multiple microphone arrays and human position information , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Khaled A. Harras,et al.  FemtoClouds Beyond the Edge: The Overlooked Data Centers , 2020, IEEE Internet of Things Magazine.

[10]  M. Tech.  Overview and Evaluation of Bluetooth Low Energy: An Emerging Low-Power Wireless Technology , 2013 .

[11]  Eike Rehder,et al.  Head detection and orientation estimation for pedestrian safety , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[12]  Rainer Stiefelhagen,et al.  Pointing gesture recognition based on 3D-tracking of face, hands and head orientation , 2003, ICMI '03.

[13]  Khaled A. Harras,et al.  Vision: The Case for Symbiosis in the Internet of Things , 2015, MCS '15.

[14]  Mohamed N. El-Derini,et al.  GAC: Energy-Efficient Hybrid GPS-Accelerometer-Compass GSM Localization , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[15]  Chulhong Min,et al.  Earables for Personal-Scale Behavior Analytics , 2018, IEEE Pervasive Computing.

[16]  Mohsen Guizani,et al.  Design Challenges of Multi-UAV Systems in Cyber-Physical Applications: A Comprehensive Survey and Future Directions , 2018, IEEE Communications Surveys & Tutorials.

[17]  Khaled A. Harras,et al.  EdgeStore: Leveraging Edge Devices for Mobile Storage Offloading , 2018, 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom).

[18]  Subramanian Ramanathan,et al.  Exploring Transfer Learning Approaches for Head Pose Classification from Multi-view Surveillance Images , 2013, International Journal of Computer Vision.

[19]  Khaled A. Harras,et al.  More Than The Sum of Its Things: Resource Sharing Across IoTs at The Edge , 2020, 2020 IEEE/ACM Symposium on Edge Computing (SEC).

[20]  Khaled A. Harras,et al.  On Realistic Target Coverage by Autonomous Drones , 2017, ACM Trans. Sens. Networks.

[21]  Pei Zhang,et al.  Polaris: getting accurate indoor orientations for mobile devices using ubiquitous visual patterns on ceilings , 2012, HotMobile '12.

[22]  Carles Gomez,et al.  Overview and Evaluation of Bluetooth Low Energy: An Emerging Low-Power Wireless Technology , 2012, Sensors.

[23]  Laurent Itti,et al.  Walking compass with head-mounted IMU sensor , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Moustafa Youssef,et al.  It's the Human that Matters: Accurate User Orientation Estimation for Mobile Computing Applications , 2014, MobiQuitous.

[25]  Khaled A. Harras,et al.  WiGest: A ubiquitous WiFi-based gesture recognition system , 2014, 2015 IEEE Conference on Computer Communications (INFOCOM).

[26]  Takahiro Okabe,et al.  Head direction estimation from low resolution images with scene adaptation , 2013, Comput. Vis. Image Underst..

[27]  Chulhong Min,et al.  Exploring audio and kinetic sensing on earable devices , 2018, WearSys@MobiSys.

[28]  Moustafa Youssef,et al.  Zephyr: Ubiquitous accurate multi-sensor fusion-based respiratory rate estimation using smartphones , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[29]  Romit Roy Choudhury,et al.  Ear-AR: indoor acoustic augmented reality on earphones , 2020, MobiCom.

[30]  Li Li,et al.  The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research , 2017, i-Perception.

[31]  Subramanian Ramanathan,et al.  A Multi-Task Learning Framework for Head Pose Estimation under Target Motion , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.