On-line human activity recognition from audio and home automation sensors: Comparison of sequential and non-sequential models in realistic Smart Homes

Automatic human Activity Recognition (AR) is an important process for the provision of context-aware services in smart spaces such as voice-controlled smart homes. In this paper, we present an on-line Activities of Daily Living (ADL) recognition method for automatic identification within homes in which multiple sensors, actuators and automation equipment coexist, including audio sensors. Three sequence-based models are presented and compared: a Hidden Markov Model (HMM), Conditional Random Fields (CRF) and a sequential Markov Logic Network (MLN). These methods have been tested in two real Smart Homes thanks to experiments involving more than 30 participants. Their results were compared to those of three non-sequential models: a Support Vector Machine (SVM), a Random Forest (RF) and a non-sequential MLN. This comparative study shows that CRF gave the best results for on-line activity recognition from non-visual, audio and home automation sensors.

[1]  Zaw Zaw Htike,et al.  Real-Time Human Activity Recognition Using External and Internal Spatial Features , 2010, 2010 Sixth International Conference on Intelligent Environments.

[2]  John Bigham,et al.  Activity recognition using a hierarchical framework , 2008, 2008 Second International Conference on Pervasive Computing Technologies for Healthcare.

[3]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[4]  Lukasz A. Kurgan,et al.  CAIM discretization algorithm , 2004, IEEE Transactions on Knowledge and Data Engineering.

[5]  Rosemarie Velik.  A brain-inspired multimodal data mining approach for human activity recognition in elderly homes , 2014, J. Ambient Intell. Smart Environ..

[6]  Alexander Artikis,et al.  Logic-based representation, reasoning and machine learning for event recognition , 2010, DEBS '10.

[7]  Heiner Stuckenschmidt,et al.  A Statistical-Relational Activity Recognition Framework for Ambient Assisted Living Systems , 2010, ISAmI.

[8]  Manuela M. Veloso,et al.  Conditional random fields for activity recognition , 2007, AAMAS '07.

[9]  Douglas A. Reynolds,et al.  Speaker Verification Using Adapted Gaussian Mixture Models , 2000, Digit. Signal Process..

[10]  Sohail Asghar,et al.  A REVIEW OF FEATURE SELECTION TECHNIQUES IN STRUCTURE LEARNING , 2013 .

[11]  Weng-Keen Wong,et al.  Logical Hierarchical Hidden Markov Models for Modeling User Activities , 2008, ILP.

[12]  Pedro Larrañaga,et al.  A review of feature selection techniques in bioinformatics , 2007, Bioinform..

[13]  Stephan Tassart,et al.  RecordMe: A Smartphone Application for Experimental Collections of Large Amount of Data Respecting Volunteer's Privacy , 2014, UCAmI.

[14]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[15]  Gwenn Englebienne,et al.  Hierarchical Activity Recognition Using Automatically Clustered Actions , 2011, AmI.

[16]  Diane J. Cook,et al.  MavHome: an agent-based smart home , 2003, Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)..

[17]  Michel Vacher,et al.  CIRDO: Smart companion for helping elderly to live at home for longer ☆ , 2014 .

[18]  P. Tang,et al.  'Smart' homes and telecare for independent living , 2000, Journal of telemedicine and telecare.

[19]  Alex Mihailidis,et al.  A Survey on Ambient-Assisted Living Tools for Older Adults , 2013, IEEE Journal of Biomedical and Health Informatics.

[20]  Francis Jambon,et al.  Une plateforme usage pour l'intégration de l'informatique ambiante dans l'habitat. L'appartement Domus , 2013, Tech. Sci. Informatiques.

[21]  Diane J. Cook,et al.  Activity recognition on streaming sensor data , 2014, Pervasive Mob. Comput..

[22]  Brigitte Meillon,et al.  Evaluation of a Context-Aware Voice Interface for Ambient Assisted Living , 2015, ACM Trans. Access. Comput..

[23]  Muhammad Raffay Hamid,et al.  A computational framework for unsupervised analysis of everyday human activities , 2008 .

[24]  Leslie Pack Kaelbling,et al.  Activity Recognition from Physiological Data using Conditional Random Fields , 2006 .

[25]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[26]  H. Yanco,et al.  Automation as Caregiver: A Survey of Issues and Technologies , 2003 .

[27]  David Minnen,et al.  Unsupervised discovery of activity primitives from multivariate sensor data , 2008 .

[28]  Pedro M. Domingos,et al.  Efficient Weight Learning for Markov Logic Networks , 2007, PKDD.

[29]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[30]  Juan Carlos Augusto,et al.  The Use of Temporal Reasoning and Management of Complex Events in Smart Homes , 2004, ECAI.

[31]  Liming Chen,et al.  Dynamic sensor data segmentation for real-time knowledge-driven activity recognition , 2014, Pervasive Mob. Comput..

[32]  François Portet,et al.  Méthodes SVM et MLN pour la reconnaissance automatique d'activités humaines dans les habitats perceptifs: tests et perspectives , 2012 .

[33]  Svetha Venkatesh,et al.  Efficient duration and hierarchical modeling for human activity recognition , 2009, Artif. Intell..

[34]  Iván Pau,et al.  The Elderly’s Independent Living in Smart Homes: A Characterization of Activities and Sensing Infrastructure Survey to Facilitate Services Development , 2015, Sensors.

[35]  Ben Taskar,et al.  Discriminative Probabilistic Models for Relational Data , 2002, UAI.

[36]  Michel Vacher,et al.  SVM-Based Multimodal Classification of Activities of Daily Living in Health Smart Homes: Sensors, Algorithms, and First Experimental Results , 2010, IEEE Transactions on Information Technology in Biomedicine.

[37]  Alexander Artikis,et al.  Behaviour Recognition from Video Content: a Logic Programming Approach , 2010, Int. J. Artif. Intell. Tools.

[38]  Francois Bremond,et al.  A Computer system to monitor older adults at home: Preliminary results , 2009 .

[39]  Gwenn Englebienne,et al.  An activity monitoring system for elderly care using generative and discriminative models , 2010, Personal and Ubiquitous Computing.

[40]  Brigitte Meillon,et al.  The sweet-home project: Audio technology in smart homes to improve well-being and reliance , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[41]  Michel Vacher,et al.  Location of an inhabitant for domotic assistance through fusion of audio and non-visual data , 2011, 2011 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops.

[42]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[43]  Stephen S. Intille,et al.  Designing a Home of the Future , 2002, IEEE Pervasive Comput..

[44]  Chris D. Nugent,et al.  A Logical Framework for Behaviour Reasoning and Assistance in a Smart Home , 2008 .

[45]  Chris D. Nugent,et al.  A Knowledge-Driven Approach to Activity Recognition in Smart Homes , 2012, IEEE Transactions on Knowledge and Data Engineering.

[46]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[47]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[48]  Luca Benini,et al.  Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection , 2008, EWSN.

[49]  David Garlan,et al.  Context is key , 2005, CACM.

[50]  Michel Vacher,et al.  Making Context Aware Decision from Uncertain Information in a Smart Home: A Markov Logic Network Approach , 2013, AmI.

[51]  Lluís A. Belanche Muñoz,et al.  Feature selection algorithms: a survey and experimental evaluation , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[52]  Martin Becker,et al.  Rule-based activity recognition framework: Challenges, technique and learning , 2009, 2009 3rd International Conference on Pervasive Computing Technologies for Healthcare.

[53]  Svetha Venkatesh,et al.  Learning Discriminative Sequence Models from Partially Labelled Data for Activity Recognition , 2008, PRICAI.

[54]  Michel Vacher,et al.  Preliminary evaluation of speech/sound recognition for telemedicine application in a real environment , 2008, INTERSPEECH.

[55]  Henry A. Kautz,et al.  Intelligent Ubiquitous Computing to Support Alzheimer ’ s Patients : Enabling the Cognitively Disabled , 2002 .

[56]  Lawrence B. Holder,et al.  Conditional random fields for activity recognition in smart environments , 2010, IHI.

[57]  Ruimin Hu,et al.  Investigating Information Search by People with Cognitive Disabilities , 2015, TACC.

[58]  Michel Vacher,et al.  Complete Sound and Speech Recognition System for Health Smart Homes: Application to the Recognition of Activities of Daily Living , 2010 .

[59]  Okko Johannes Räsänen.  Hierarchical unsupervised discovery of user context from multivariate sensory data , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[60]  Tae-Seong Kim,et al.  A single tri-axial accelerometer-based real-time personal life log system capable of human activity recognition and exercise information generation , 2011, Personal and Ubiquitous Computing.

[61]  Jacques Demongeot,et al.  A model for the measurement of patient activity in a hospital suite , 2006, IEEE Transactions on Information Technology in Biomedicine.

[62]  Brigitte Meillon,et al.  Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects , 2011, Personal and Ubiquitous Computing.

[63]  Michael C. Mozer,et al.  Lessons from an Adaptive Home , 2005 .

[64]  Brigitte Meillon,et al.  The Sweet-Home speech and multimodal corpus for home automation interaction , 2014, LREC.

[65]  Heiner Stuckenschmidt,et al.  Recognizing interleaved and concurrent activities: A statistical-relational approach , 2011, 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[66]  Naveen Vignesh Ramaraj.  Location Based Activity Recognition Using Mobile Phones , 2009 .

[67]  Maryam Mahdaviani.  Semi-supervised and active training of conditional random fields for activity recognition , 2007 .

[68]  Henry A. Kautz,et al.  Sensor-Based Understanding of Daily Life via Large-Scale Use of Common Sense , 2006, AAAI.

[69]  Michel Vacher,et al.  A french corpus of audio and multimodal interactions in a health smart home , 2012, Journal on Multimodal User Interfaces.

[70]  B. Kröse,et al.  Bayesian Activity Recognition in Residence for Elders , 2007 .

[71]  Monique Thonnat,et al.  Activity Recognition from Video Sequences using Declarative Models , 2000, ECAI.

[72]  Alexander Artikis,et al.  Logic-based event recognition , 2012, The Knowledge Engineering Review.

[73]  Ben Taskar,et al.  Introduction to statistical relational learning , 2007 .

[74]  M. Lawton,et al.  Assessment of Older People: Self-Maintaining and Instrumental Activities of Daily Living , 1969 .

[75]  Mark R. Segal,et al.  Machine Learning Benchmarks and Random Forest Regression , 2004 .

[76]  Radha Poovendran,et al.  Activity Recognition Using a Combination of Category Components and Local Models for Video Surveillance , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[77]  Tae-Seong Kim,et al.  A Triaxial Accelerometer-Based Physical-Activity Recognition via Augmented-Signal Features and a Hierarchical Recognizer , 2010, IEEE Transactions on Information Technology in Biomedicine.

[78]  Michel Vacher,et al.  Improving Supervised Classification of Activities of Daily Living Using Prior Knowledge , 2011, Int. J. E Health Medical Commun..

[79]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[80]  Qiang Ji,et al.  Learning dynamic Bayesian network discriminatively for human activity recognition , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[81]  L. De Raedt,et al.  Logical Hidden Markov Models , 2011, J. Artif. Intell. Res..

[82]  Johan Himberg,et al.  Unsupervised clustering of symbol strings and context recognition , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[83]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[84]  Berardina De Carolis,et al.  C@sa: Intelligent Home Control and Simulation , 2004, International Conference on Computational Intelligence.

[85]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[86]  Donald E. Brown,et al.  A Rule-Based Approach to the Analysis of Elders' Activity Data: Detection of Health and Possible Emergency Conditions , 2005, AAAI Fall Symposium: Caring Machines.

[87]  Brigitte Meillon,et al.  The Sweet-Home project: Audio processing and decision making in smart home to improve well-being and reliance , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[88]  Rong Chen,et al.  Latent-Dynamic Conditional Random Fields for recognizing activities in smart homes , 2014, J. Ambient Intell. Smart Environ..

[89]  Brian Patrick Clarkson,et al.  Life patterns : structure from wearable sensors , 2002 .

[90]  Sydney Katz.  Assessing Self‐maintenance: Activities of Daily Living, Mobility, and Instrumental Activities of Daily Living , 1983, Journal of the American Geriatrics Society.

[91]  Michel Vacher,et al.  Information extraction from sound for medical telemonitoring , 2006, IEEE Transactions on Information Technology in Biomedicine.