Combining multiple sensors for event recognition of older people

We herein present a hierarchical model-based framework for event recognition using multiple sensors. Event models combine a priori knowledge of the scene (3D geometric and semantic information, such as contextual zones and equipment) with moving objects (e.g., a Person) detected by a monitoring system. The event models follow a generic ontology based on natural language, which allows domain experts to easily adapt them. The framework novelty relies on combining multiple sensors at decision (event) level, and handling their conflict using a probabilistic approach. The proposed approach for event conflict handling computes the event reliability for each sensor, and then combines them using Dempster-Shafer Theory with an alternative combination rule. The proposed framework is evaluated using multi-sensor recording of instrumental daily living activities (e.g., watching TV, writing a check, preparing tea, organizing week intake of prescribed medication) of participants of a clinical trial for Alzheimer's disease. Two evaluation cases are presented: the combination of events (or activities) from heterogeneous sensors (RGB ambient camera and a wearable inertial sensor) by a deterministic fashion, and the combination of conflicting events recognized by video cameras with partially overlapped field of view (a RGB- and a RGB-D-camera, Kinect®). The results show the framework improves the event recognition rate in both cases.

[1]  François Brémond,et al.  An Activity Monitoring System for Real Elderly at Home: Validation Study , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[2]  Ehud Rivlin,et al.  Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[3]  Bernadette Dorizzi,et al.  A pervasive multi-sensor data fusion for smart home healthcare monitoring , 2011, 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011).

[4]  Michel Vacher,et al.  Introducing knowledge in the process of supervised classification of activities of Daily Living in Health Smart Homes , 2010, The 12th IEEE International Conference on e-Health Networking, Applications and Services.

[5]  James F. Allen Maintaining knowledge about temporal intervals , 1983, CACM.

[6]  Martial Hebert,et al.  Activity Forecasting , 2012, ECCV.

[7]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[8]  Shan Gao,et al.  Performance evaluation of early and late fusion methods for generic semantics indexing , 2013, Pattern Analysis and Applications.

[9]  François Brémond,et al.  A Generic Framework for Video Understanding Applied to Group Behavior Recognition , 2012, 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance.

[10]  Quoc V. Le,et al.  Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis , 2011, CVPR 2011.

[11]  François Brémond,et al.  A Framework Dealing with Uncertainty for Complex Event Recognition , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[12]  François Brémond,et al.  Automatic Video Interpretation: A Novel Algorithm for Temporal Scenario Recognition , 2003, IJCAI.

[13]  Zhang Hong-cai CONFLICT PROBLEM OF DEMPSTER-SHAFER EVIDENCE THEORY , 2001 .

[14]  Rong Liu,et al.  Recognizing Human Activities Based on Multi-Sensors Fusion , 2010, 2010 4th International Conference on Bioinformatics and Biomedical Engineering.

[15]  M. Folstein,et al.  The Mini-Mental State Examination. , 1983, Archives of general psychiatry.

[16]  Linmi Tao,et al.  An Event-driven Context Model in Elderly Health Monitoring , 2009, 2009 Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing.

[17]  Yi Yang,et al.  E-LAMP: integration of innovative ideas for multimedia event detection , 2013, Machine Vision and Applications.

[18]  Monique Thonnat,et al.  Controlling background subtraction algorithms for robust object detection , 2009, ICDP.

[19]  Duc Phu Chau,et al.  A multi-feature tracking algorithm enabling adaptation to context variations , 2011, ICDP.

[20]  Mubarak Shah,et al.  Recognizing Complex Events Using Large Margin Joint Low-Level Event Model , 2012, ECCV.

[21]  F. Bremond,et al.  A Multi-Sensor Approach for Activity Recognition in Older Patients , 2012 .

[22]  A. G. Amitha Perera,et al.  Multimedia event detection with multimodal feature fusion and temporal concept localization , 2013, Machine Vision and Applications.

[23]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[24]  Lei Gao,et al.  A system for activity recognition using multi-sensor fusion , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[25]  Ramakant Nevatia,et al.  Evaluating multimedia features and fusion for example-based event detection , 2013, Machine Vision and Applications.

[26]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[27]  Dong Liu,et al.  Discovering joint audio–visual codewords for video event detection , 2013, Machine Vision and Applications.