A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning

Recent advances in Internet of Things (IoT) technologies and the reduction in the cost of sensors have encouraged the development of smart environments, such as smart homes. Smart homes can offer home assistance services to improve the quality of life, autonomy, and health of their residents, especially for the elderly and dependent. To provide such services, a smart home must be able to understand the daily activities of its residents. Techniques for recognizing human activity in smart homes are advancing daily. However, new challenges are emerging every day. In this paper, we present recent algorithms, works, challenges, and taxonomy of the field of human activity recognition in a smart home through ambient sensors. Moreover, since activity recognition in smart homes is a young field, we raise specific problems, as well as missing and needed contributions. However, we also propose directions, research opportunities, and solutions to accelerate advances in this field.

[1]  Ivan Marsic,et al.  Deep Learning for RFID-Based Activity Recognition , 2016, SenSys.

[2]  Mohamed Sedky,et al.  OpenSHS: Open Smart Home Simulator , 2017, Sensors.

[3]  Diane J. Cook,et al.  Learning Setting-Generalized Activity Models for Smart Spaces , 2012, IEEE Intelligent Systems.

[4]  Sydney Katz.  Assessing Self‐maintenance: Activities of Daily Living, Mobility, and Instrumental Activities of Daily Living , 1983, Journal of the American Geriatrics Society.

[5]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[6]  Alaa Alhamoud,et al.  Activity Recognition in Multi-User Environments Using Techniques of Multi-label Classification , 2016, IOT.

[7]  Henry A. Kautz,et al.  Inferring activities from interactions with objects , 2004, IEEE Pervasive Computing.

[8]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[9]  Guilin Chen,et al.  Human Activity Recognition in a Smart Home Environment with Stacked Denoising Autoencoders , 2016, WAIM Workshops.

[10]  Diane J. Cook,et al.  Activity Discovery and Activity Recognition: A New Partnership , 2013, IEEE Transactions on Cybernetics.

[11]  Ian Craddock,et al.  A dataset for room level indoor localization using a smart home in a box , 2019, Data in brief.

[12]  Amos Storkey,et al.  Meta-Learning in Neural Networks: A Survey , 2020, IEEE transactions on pattern analysis and machine intelligence.

[13]  Germain Forestier,et al.  Transfer learning for time series classification , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[14]  Yunqian Ma,et al.  Imbalanced Learning: Foundations, Algorithms, and Applications , 2013 .

[15]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[16]  Chris D. Nugent,et al.  Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition , 2018, Expert Syst. Appl..

[17]  Diane J. Cook,et al.  CASAS: A Smart Home in a Box , 2013, Computer.

[18]  Vladlen Koltun,et al.  Playing for Benchmarks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Chris D. Nugent,et al.  UCAmI Cup. Analyzing the UJA Human Activity Recognition Dataset of Activities of Daily Living , 2018, UCAmI.

[20]  Lina Yao,et al.  Deep Learning for Sensor-based Human Activity Recognition , 2021, ACM Comput. Surv..

[21]  Ivan Marsic,et al.  Concurrent Activity Recognition with Multimodal CNN-LSTM Structure , 2017, ArXiv.

[22]  Lei He,et al.  Human activity recognition based on feature selection in smart home using back-propagation algorithm. , 2014, ISA transactions.

[23]  Francisco Herrera,et al.  Learning from Imbalanced Data Sets , 2018, Springer International Publishing.

[24]  Chris D. Nugent,et al.  A Logical Framework for Behaviour Reasoning and Assistance in a Smart Home , 2008 .

[25]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[26]  C A Nelson,et al.  Learning to Learn , 2017, Encyclopedia of Machine Learning and Data Mining.

[27]  Chris D. Nugent,et al.  Human Activity Recognition and Behaviour Analysis , 2019, Springer International Publishing.

[28]  Satoshi Tanaka,et al.  Applying Ontology and Probabilistic Model to Human Activity Recognition from Surrounding Things , 2007 .

[29]  LuKun Wang,et al.  Human Activity Recognition Based on Wearable Sensor Using Hierarchical Deep LSTM Networks , 2019, Circuits, Systems, and Signal Processing.

[30]  L. Whitmarsh,et al.  Social barriers to the adoption of smart homes , 2013 .

[31]  Maxime Devanne,et al.  Recognition of Activities of Daily Living via Hierarchical Long-Short Term Memory Networks , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[32]  Wai Lok Woo,et al.  Joint Learning of Temporal Models to Handle Imbalanced Data for Human Activity Recognition , 2020, Applied Sciences.

[33]  Mohamed-Rafik Bouguelia,et al.  Efficient Activity Recognition in Smart Homes Using Delayed Fuzzy Temporal Windows on Binary Sensors , 2020, IEEE Journal of Biomedical and Health Informatics.

[34]  Christophe Lohr,et al.  Fully Convolutional Network Bootstrapped by Word Encoding and Embedding for Activity Recognition in Smart Homes , 2020, Communications in Computer and Information Science.

[35]  Anthony G. Cohn,et al.  Feature Space Analysis for Human Activity Recognition in Smart Environments , 2016, 2016 12th International Conference on Intelligent Environments (IE).

[36]  L. Minh Dang,et al.  Sensor-based and vision-based human activity recognition: A comprehensive survey , 2020, Pattern Recognit..

[37]  Sohail Sarwar,et al.  Ontology-driven semantic unified modelling for concurrent activity recognition (OSCAR) , 2018, Multimedia Tools and Applications.

[38]  George Moore,et al.  A Comparative Analysis of Windowing Approaches in Dense Sensing Environments , 2018, UCAmI.

[39]  Kyungeun Cho,et al.  Persim 3D: Context-Driven Simulation and Modeling of Human Activities in Smart Spaces , 2015, IEEE Transactions on Automation Science and Engineering.

[40]  Chris D. Nugent,et al.  Ontology-based activity recognition in intelligent pervasive environments , 2009, Int. J. Web Inf. Syst..

[41]  Parviz Asghari,et al.  Online human activity recognition employing hierarchical hidden Markov models , 2019, J. Ambient Intell. Humaniz. Comput..

[42]  Jesse Hoey,et al.  Activity Recognition in Pervasive Intelligent Environments , 2011 .

[43]  Andreas Holzinger,et al.  Users' Perceptions and Attitudes Towards Smart Home Technologies , 2018, ICOST.

[44]  Manuela M. Veloso,et al.  Conditional random fields for activity recognition , 2007, AAMAS '07.

[45]  Matjaz Gams,et al.  Competitive Live Evaluations of Activity-Recognition Systems , 2015, IEEE Pervasive Computing.

[46]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2019, Neural Networks.

[47]  Nawel Yala,et al.  Feature extraction for human activity recognition on streaming data , 2015, 2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA).

[48]  Chris D. Nugent,et al.  Evidential fusion of sensor data for activity recognition in smart homes , 2009, Pervasive Mob. Comput..

[49]  Wing W. Y. Ng,et al.  Neural Network Ensembles for Sensor-Based Human Activity Recognition Within Smart Environments , 2019, Sensors.

[50]  Andres Mendez-Vazquez,et al.  Simulating Events to Generate Synthetic Data for Pervasive Spaces , 2008 .

[51]  Chris D. Nugent,et al.  The creation of simulated activity datasets using a graphical intelligent environment simulation tool , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[52]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[53]  Chris D. Nugent,et al.  Semantic Smart Homes: Towards Knowledge Rich Assisted Living Environments , 2009 .

[54]  R. Stiefelhagen,et al.  Let’s Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[55]  Jake K. Aggarwal,et al.  Human activity recognition from 3D data: A review , 2014, Pattern Recognit. Lett..

[56]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[57]  Heinrich C. Mayr,et al.  A windowing approach for activity recognition in sensor data streams , 2016, 2016 Eighth International Conference on Ubiquitous and Future Networks (ICUFN).

[58]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  B. Kröse,et al.  Bayesian Activity Recognition in Residence for Elders , 2007 .

[60]  Qing Zhang,et al.  Multi-Resident Activity Monitoring in Smart Homes: A Case Study , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[61]  Özlem Durmaz Incel,et al.  ARAS human activity datasets in multiple homes with multiple residents , 2013, 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops.

[62]  Tan-Hsu Tan,et al.  Unobtrusive Activity Recognition of Elderly People Living Alone Using Anonymous Binary Sensors and DCNN , 2019, IEEE Journal of Biomedical and Health Informatics.

[63]  James L. Crowley,et al.  A Dataset of Routine Daily Activities in an Instrumented Home , 2017, UCAmI.

[64]  T. V. Kasteren,et al.  Activity recognition for health monitoring elderly using temporal probabilistic models , 2007 .

[65]  Wai Lok Woo,et al.  Dilated causal convolution with multi-head self attention for sensor human activity recognition , 2021, Neural Computing and Applications.

[66]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[67]  Gwenn Englebienne,et al.  Human activity recognition from wireless sensor network data: benchmark and software , 2011 .

[68]  Macarena Espinilla,et al.  Sensor-Based Datasets for Human Activity Recognition – A Systematic Review of Literature , 2018, IEEE Access.

[69]  Eric Campo,et al.  A review of smart homes - Present state and future challenges , 2008, Comput. Methods Programs Biomed..

[70]  Chris D. Nugent,et al.  A Knowledge-Driven Approach to Activity Recognition in Smart Homes , 2012, IEEE Transactions on Knowledge and Data Engineering.

[71]  Ahmad Lotfi,et al.  Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data , 2020, PETRA.

[72]  Diane J. Cook,et al.  Transfer learning for activity recognition: a survey , 2013, Knowledge and Information Systems.

[73]  Matthieu Geist,et al.  Human Activity Recognition Using Recurrent Neural Networks , 2017, CD-MAKE.

[74]  Matthai Philipose,et al.  Mining models of human activities from the web , 2004, WWW '04.

[75]  Oliver Brdiczka,et al.  Learning Situation Models in a Smart Home , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[76]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[77]  Mohamed Sedky,et al.  Evaluating Machine Learning Techniques for Activity Classification in Smart Home Environments , 2018 .

[78]  Chih-Yung Chang,et al.  Activities of Daily Living Recognition With Binary Environment Sensors Using Deep Learning: A Comparative Study , 2021, IEEE Sensors Journal.

[79]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[80]  Emanuele Frontoni,et al.  A sequential deep learning application for recognising human activities in smart homes , 2020, Neurocomputing.

[81]  Tan-Hsu Tan,et al.  Multi-Resident Activity Recognition in a Smart Home Using RGB Activity Image and DCNN , 2018, IEEE Sensors Journal.

[82]  Anthony G. Cohn,et al.  Learning Hierarchical Models of Complex Daily Activities from Annotated Videos , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[83]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[84]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[85]  Diane J Cook,et al.  Assessing the Quality of Activities in a Smart Environment , 2009, Methods of Information in Medicine.

[86]  Diane J. Cook,et al.  Activity2Vec: Learning ADL Embeddings from Sensor Data with a Sequence-to-Sequence Model , 2019, ArXiv.

[87]  Christian Igel,et al.  U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging , 2019, NeurIPS.

[88]  Diane J. Cook,et al.  Enhancing activity recognition using CPD-based activity segmentation , 2019, Pervasive Mob. Comput..

[89]  Diane J. Cook,et al.  Activity recognition on streaming sensor data , 2014, Pervasive Mob. Comput..

[90]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[91]  Matthieu Geist,et al.  Convolutional and Recurrent Neural Networks for Activity Recognition in Smart Environment , 2015, BIRS-IMLKE.

[92]  Diane Myung-kyung Woodbridge,et al.  Sensor Selection for Activity Classification at Smart Home Environments , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[93]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[94]  Thinagaran Perumal,et al.  Sequential neural networks for multi-resident activity recognition in ambient sensing smart homes , 2021, Applied Intelligence.

[95]  知秀 柴田.  5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding , 2020 .

[96]  Shu-Ching Chen,et al.  Multimedia Big Data Analytics , 2018, ACM Comput. Surv..

[97]  J. C. Schlimmer,et al.  Incremental learning from noisy data , 2004, Machine Learning.

[98]  Diane J. Cook,et al.  Persim - Simulator for Human Activities in Pervasive Spaces , 2011, 2011 Seventh International Conference on Intelligent Environments.

[99]  Ioannis A. Kakadiaris,et al.  A Review of Human Activity Recognition Methods , 2015, Front. Robot. AI.

[100]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[101]  Dominique Duhaut,et al.  Intrinsically Motivated Open-Ended Multi-Task Learning Using Transfer Learning to Discover Task Hierarchy , 2021, Applied Sciences.

[102]  Gregory D. Abowd,et al.  Developing shared home behavior datasets to advance HCI and ubiquitous computing research , 2009, CHI Extended Abstracts.

[103]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[104]  Quan Z. Sheng,et al.  Different Approaches for Human Activity Recognition: A Survey , 2019, ArXiv.

[105]  Michal Koperski,et al.  Toyota Smarthome: Real-World Activities of Daily Living , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[106]  Araceli Sanchis,et al.  Activity Recognition Using Hybrid Generative/Discriminative Models on Home Environments Using Binary Sensors , 2013, Sensors.

[107]  Daniel Retkowitz,et al.  Simulation of Smart Environments , 2007, IEEE International Conference on Pervasive Services.

[108]  Sung-Bong Yang,et al.  Deep neural networks for activity recognition with multi-sensor data in a smart home , 2018, 2018 IEEE 4th World Forum on Internet of Things (WF-IoT).

[109]  Abdenour Hadid,et al.  Vision-based human activity recognition: a survey , 2020, Multimedia Tools and Applications.

[110]  Chris D. Nugent,et al.  Simulation of Smart Home Activity Datasets , 2015, Sensors.

[111]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[112]  Luis Gomes,et al.  An Intelligent Smart Plug with Shared Knowledge Capabilities , 2018, Sensors.

[113]  Jennifer Healey,et al.  A Long-Term Evaluation of Sensing Modalities for Activity Recognition , 2007, UbiComp.

[114]  Chen Yu,et al.  A configurable context-aware simulator for smart home systems , 2011, 2011 6th International Conference on Pervasive Computing and Applications.

[115]  Michel Vacher,et al.  SVM-Based Multimodal Classification of Activities of Daily Living in Health Smart Homes: Sensors, Algorithms, and First Experimental Results , 2010, IEEE Transactions on Information Technology in Biomedicine.

[116]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[117]  Kwei-Jay Lin,et al.  Using Latent Knowledge to Improve Real-Time Activity Recognition for Smart IoT , 2020, IEEE Transactions on Knowledge and Data Engineering.