Multi-modal activity recognition from egocentric vision, semantic enrichment and lifelogging applications for the care of dementia

Abstract We describe a framework for lifelogging monitoring in the scope of dementia care, based on activity recognition from egocentric vision and semantic context-enrichment. As pure vision-based approaches appear to be already saturating in terms of recognition accuracy, we propose their enhancement with wearable bracelet accelerometer information. For that purpose, we design and study appropriate early and late fusion schemes to increase accuracy. The incorporation of mechanical variables, such as jerk, improves the recognition accuracy of activities that require fine motion. In addition, we describe a framework for semantic activity representation and interpretation, using Semantic Web technologies for building interoperable activity graphs. The system is personalized, as deployment-specific activity models are authored, while problems related to the disease are detected by rules. Complemented by lifelogging applications, the system is able to support interventions by clinicians, and endorse a feeling of safety and inclusion for end-users and their carers.