Human behavior sensing: challenges and approaches

In recent years, Activities of Daily Living Scale (ADLs) is widely used to evaluate living abilities of the patients and the elderly. So, the study of behavior sensing has attracted more and more attention of researchers. Behavior sensing technology is of strong theoretical and practical value in the fields of smart home and virtual reality. Most of the currently proposed approaches for tracking indicators of ADLs are human-centric, which classify activities using physical information of the observed persons. Considering the privacy concerns of the human-centric approaches (e.g. images of home environment, private behavior), researchers have also proposed some thing-centric approaches, which use environmental information on things (e.g. the vibration of things) to infer human activity. In this paper, by considering the unified steps in both the human-centric approaches and the thing-centric approaches, we make a comprehensive survey on the challenges and proposed methods to do behavior sensing, which are signal collection, preprocessing, feature extraction, and activity recognition. Moreover, based on the latest research progress, we post a perspective from our standpoint, discussing future outlook and challenges of human behavior sensing.

[1]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[2]  Arthur D. Fisk,et al.  Benefits and Privacy Concerns of a Home Equipped with a Visual Sensing System: A Perspective from Older Adults , 2006 .

[3]  Zhen Liu,et al.  A comparison of improving multi-class imbalance for internet traffic classification , 2014, Inf. Syst. Frontiers.

[4]  Jun Luo,et al.  CeilingSee: Device-free occupancy inference through lighting infrastructure based LED sensing , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[5]  Siyu Jiang,et al.  Whole-home gesture recognition using wireless signals (demo) , 2013, SIGCOMM.

[6]  Peter H. N. de With,et al.  Automatic video-based human motion analyzer for consumer surveillance system , 2009, IEEE Transactions on Consumer Electronics.

[7]  Giuseppe Aceto,et al.  Mobile Encrypted Traffic Classification Using Deep Learning: Experimental Evaluation, Lessons Learned, and Challenges , 2019, IEEE Transactions on Network and Service Management.

[8]  Antonio Pescapè,et al.  Know your Big Data Trade-offs when Classifying Encrypted Mobile Traffic with Deep Learning , 2019, 2019 Network Traffic Measurement and Analysis Conference (TMA).

[9]  Massimo Fornasier,et al.  Low-rank Matrix Recovery via Iteratively Reweighted Least Squares Minimization , 2010, SIAM J. Optim..

[10]  Guoliang Xing,et al.  iSleep: unobtrusive sleep quality monitoring using smartphones , 2013, SenSys '13.

[11]  Antonio Pescapè,et al.  Multi-classification approaches for classifying mobile app traffic , 2018, J. Netw. Comput. Appl..

[12]  Haipeng Dai,et al.  UltraGesture: Fine-Grained Gesture Sensing and Recognition , 2018, 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON).

[13]  Motoki Miura,et al.  A Study of Long Term Tendencies in Residents' Activities of Daily Living at a Group Home for People with Dementia Using RFID Slippers , 2011, ICOST.

[14]  Ted Herman,et al.  Hand hygiene duration and technique recognition using wrist-worn sensors , 2015, IPSN.

[15]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[16]  Araceli Sanchis,et al.  Activity Recognition Using Hybrid Generative/Discriminative Models on Home Environments Using Binary Sensors , 2013, Sensors.

[17]  Antonio Pescapè,et al.  TaRad: A Thing-Centric Sensing System for Detecting Activities of Daily Living , 2019, IDCS.

[18]  Hojung Cha,et al.  Non-obstructive room-level locating system in home environments using activity fingerprints from smartwatch , 2015, UbiComp.

[19]  Koichi Shinoda,et al.  Spectral Graph Skeletons for 3D Action Recognition , 2014, ACCV.

[20]  Mani Srivastava,et al.  Human-centric sensing , 2012, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[21]  Bastien Lyonnet,et al.  Human gait classification using microDoppler time-frequency signal representations , 2010, 2010 IEEE Radar Conference.

[22]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[23]  Wei Wang,et al.  Keystroke Recognition Using WiFi Signals , 2015, MobiCom.

[24]  Zhu Wang,et al.  FreeSense: human-behavior understanding using Wi-Fi signals , 2018, J. Ambient Intell. Humaniz. Comput..

[25]  Mohamed Ibrahim,et al.  Eyelight: Light-and-Shadow-Based Occupancy Estimation and Room Activity Recognition , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[26]  PorikliFatih,et al.  Going deeper into action recognition , 2017 .

[27]  Maria E. Niessen,et al.  Monitoring Activities of Daily Living in Smart Homes: Understanding human behavior , 2016, IEEE Signal Processing Magazine.

[28]  Guoliang Xing,et al.  PBN: towards practical activity recognition using smartphone-based body sensor networks , 2011, SenSys.

[29]  Moustafa Youssef,et al.  Robust WLAN Device-free Passive motion detection , 2012, 2012 IEEE Wireless Communications and Networking Conference (WCNC).

[30]  Sheng Tan,et al.  WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition , 2016, MobiHoc.

[31]  Jaeyoung Yang,et al.  Activity recognition based on RFID object usage for smart mobile devices , 2011 .

[32]  Petia Radeva,et al.  Human Activity Recognition from Accelerometer Data Using a Wearable Device , 2011, IbPRIA.

[33]  Xing Zhou,et al.  WiStep , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[34]  Chris D. Nugent,et al.  A sensor and video based ontology for activity recognition in smart environments , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[35]  Shyamnath Gollakota,et al.  Bringing Gesture Recognition to All Devices , 2014, NSDI.

[36]  Marc Berenguer,et al.  Non-invasive monitoring of the activities of daily living of elderly people at home – a pilot study of the usage of domestic appliances , 2008, Journal of telemedicine and telecare.

[37]  Yang Xu,et al.  WiFinger: talk to your smart devices with finger-grained gesture , 2016, UbiComp.

[38]  Kaishun Wu,et al.  WiFall: Device-free fall detection by wireless networks , 2017, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[39]  Moustafa Youssef,et al.  New insights into wifi-based device-free localization , 2013, UbiComp.

[40]  Jie Yang,et al.  E-eyes: device-free location-oriented activity identification using fine-grained WiFi signatures , 2014, MobiCom.

[41]  Miguel Á. Carreira-Perpiñán,et al.  OBSERVE: Occupancy-based system for efficient reduction of HVAC energy , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[42]  Yunhao Liu,et al.  Towards omnidirectional passive human detection , 2013, 2013 Proceedings IEEE INFOCOM.

[43]  John A. Stankovic,et al.  Empath: a continuous remote emotional health monitoring system for depressive illness , 2011, Wireless Health.

[44]  Alex Mihailidis,et al.  Feature-level combination of skeleton joints and body parts for accurate aggressive and agitated behavior recognition , 2017, J. Ambient Intell. Humaniz. Comput..

[45]  Yusheng Ji,et al.  RF-Based device-free recognition of simultaneously conducted activities , 2013, UbiComp.

[46]  Jaeseok Yun,et al.  Human Movement Detection and Idengification Using Pyroelectric Infrared Sensors , 2014, Sensors.

[47]  Xu Chen,et al.  Tracking Vital Signs During Sleep Leveraging Off-the-shelf WiFi , 2015, MobiHoc.

[48]  Mehrtash Tafazzoli Harandi,et al.  Going deeper into action recognition: A survey , 2016, Image Vis. Comput..

[49]  Panlong Yang,et al.  TW-See: Human Activity Recognition Through the Wall With Commodity Wi-Fi Devices , 2019, IEEE Transactions on Vehicular Technology.

[50]  Cem Ersoy,et al.  Multi-resident activity tracking and recognition in smart environments , 2017, Journal of Ambient Intelligence and Humanized Computing.

[51]  Paolo Spagnolo,et al.  Human Behavior Understanding in Networked Sensing , 2014, Springer International Publishing.

[52]  Timothy J. O'Shea,et al.  Semi-supervised radio signal identification , 2016, 2017 19th International Conference on Advanced Communication Technology (ICACT).

[53]  Shaojie Tang,et al.  Electronic frog eye: Counting crowd using WiFi , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[54]  M. Lawton,et al.  Assessment of older people: self-maintaining and instrumental activities of daily living. , 1969, The Gerontologist.

[55]  Antonio Pescapè,et al.  Wi-CR: Human Action Counting and Recognition with Wi-Fi Signals , 2019, 2019 4th International Conference on Computing, Communications and Security (ICCCS).

[56]  Alex Mihailidis,et al.  Aggressive and agitated behavior recognition from accelerometer data using non-negative matrix factorization , 2018, J. Ambient Intell. Humaniz. Comput..

[57]  Koji Yatani,et al.  BodyScope: a wearable acoustic sensor for activity recognition , 2012, UbiComp.

[58]  Parth H. Pathak,et al.  WiWho: WiFi-Based Person Identification in Smart Spaces , 2016, 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[59]  Moustafa Youssef,et al.  Multi-entity device-free WLAN localization , 2012, 2012 IEEE Global Communications Conference (GLOBECOM).

[60]  Ivan Marsic,et al.  Deep Learning for RFID-Based Activity Recognition , 2016, SenSys.

[61]  Neal Patwari,et al.  See-Through Walls: Motion Tracking Using Variance-Based Radio Tomography Networks , 2011, IEEE Transactions on Mobile Computing.

[62]  Yunhao Liu,et al.  Non-Invasive Detection of Moving and Stationary Human With WiFi , 2015, IEEE Journal on Selected Areas in Communications.

[63]  Rob Miller,et al.  3D Tracking via Body Radio Reflections , 2014, NSDI.

[64]  Wei Wang,et al.  Understanding and Modeling of WiFi Signal Based Human Activity Recognition , 2015, MobiCom.

[65]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.