A multisource fusion framework driven by user-defined knowledge for egocentric activity recognition

Recently, egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its widespread applicability to human systems, including the evaluation of dietary and physical activity and the monitoring of patients and older adults. In this paper, we present a knowledge-driven multisource fusion framework for the recognition of egocentric activities in daily living (ADL). This framework employs Dezert–Smarandache theory across three information sources: the wearer’s knowledge, images acquired by a wearable camera, and sensor data from wearable inertial measurement units and GPS. A simple likelihood table is designed to provide routine ADL information for each individual. A well-trained convolutional neural network is then used to produce a set of textual tags that, along with routine information and other sensor data, are used to recognize ADLs based on information theory-based statistics and a support vector machine. Our experiments show that the proposed method accurately recognizes 15 predefined ADL classes, including a variety of sedentary activities that have previously been difficult to recognize. When applied to real-life data recorded using a self-constructed wearable device, our method outperforms previous approaches, and an average accuracy of 85.4% is achieved for the 15 ADLs.

[1]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[2]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[3]  Mingui Sun,et al.  Segmentation for efficient browsing of chronical video recorded by a wearable device , 2010, Proceedings of the 2010 IEEE 36th Annual Northeast Bioengineering Conference (NEBEC).

[4]  Li Kuang,et al.  A Parallel Fusion Method for Heterogeneous Multi-sensor Transportation Data , 2011, MDAI.

[5]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[6]  James M. Rehg,et al.  Learning to Recognize Daily Actions Using Gaze , 2012, ECCV.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Chris D. Nugent,et al.  A Logical Framework for Behaviour Reasoning and Assistance in a Smart Home , 2008 .

[9]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.

[10]  Sei Naito,et al.  An Attention-Based Activity Recognition for Egocentric Video , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Edward A. Fox,et al.  Research Contributions , 2014 .

[12]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[13]  Petia Radeva,et al.  Toward Storytelling From Visual Lifelogging: An Overview , 2015, IEEE Transactions on Human-Machine Systems.

[14]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Ramakant Nevatia,et al.  VERL: An Ontology Framework for Representing and Annotating Video Events , 2005, IEEE Multim..

[17]  Jiaul H. Paik.  A novel TF-IDF weighting scheme for effective ranking , 2013, SIGIR.

[18]  Zhen Li,et al.  Daily life event segmentation for lifestyle evaluation based on multi-sensor data recorded by a wearable device , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[19]  Alan F. Smeaton,et al.  Proceedings of the 23rd ACM international conference on Multimedia , 2015, MM 2015.

[20]  Luc Van Gool,et al.  European conference on computer vision (ECCV) , 2006, eccv 2006.

[21]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[22]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[23]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[24]  Alexander G. Hauptmann,et al.  Multi-camera Egocentric Activity Detection for Personal Assistant , 2013, MMM.

[25]  Joo-Hwee Lim,et al.  Multimodal Multi-Stream Deep Learning for Egocentric Activity Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Florentin Smarandache,et al.  Advances and Applications of DSmT for Information Fusion , 2004 .

[27]  Jie Lin,et al.  Egocentric activity recognition with multimodal fisher vector , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Takahiro Okabe,et al.  Fast unsupervised ego-action learning for first-person sports videos , 2011, CVPR 2011.

[29]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[30]  Sophia Antipolis,et al.  FUSION FRAMEWORK FOR VIDEO EVENT RECOGNITION , 2013 .

[31]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[32]  Matthias Rauterberg,et al.  The Evolution of First Person Vision Methods: A Survey , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Li-Ping Jing,et al.  Improved feature selection approach TFIDF in text mining , 2002, Proceedings. International Conference on Machine Learning and Cybernetics.

[34]  Marinka Zitnik,et al.  Data Fusion by Matrix Factorization , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Fabien Lagriffoul,et al.  Activity Recognition Using an Egocentric Perspective of Everyday Objects , 2007, UIC.

[36]  James M. Rehg,et al.  Delving into egocentric actions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Shmuel Peleg,et al.  An Egocentric Look at Video Photographer Identity , 2014, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Abdenour Bouzouane,et al.  A Smart Home Agent for Plan Recognition of Cognitively-impaired Patients , 2006, J. Comput..

[39]  Glenn Shafer,et al.  Perspectives on the theory and practice of belief functions , 1990, Int. J. Approx. Reason..

[40]  Petia Radeva,et al.  Object Discovery Using CNN Features in Egocentric Videos , 2015, IbPRIA.

[41]  Yiran Chen,et al.  eButton: A wearable computer for health monitoring and personal assistance , 2014, 2014 51st ACM/EDAC/IEEE Design Automation Conference (DAC).

[42]  Kris M. Kitani,et al.  Going Deeper into First-Person Activity Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Tae-Seong Kim,et al.  A Triaxial Accelerometer-Based Physical-Activity Recognition via Augmented-Signal Features and a Hierarchical Recognizer , 2010, IEEE Transactions on Information Technology in Biomedicine.

[44]  Fabio Ramos,et al.  Multi-scale Conditional Random Fields for first-person activity recognition on elders and disabled patients , 2015 .

[45]  Matthieu Cornec,et al.  An entropy-based term weighting scheme and its application in e-commerce search engines , 2015 .

[46]  Jean-Christophe Nebel,et al.  Recognition of Activities of Daily Living with Egocentric Vision: A Review , 2016, Sensors.

[47]  Chong Li,et al.  A Hierarchical Deep Fusion Framework for Egocentric Activity Recognition using a Wearable Hybrid Sensor System , 2019, Sensors.

[48]  Anthony G. Cohn,et al.  Egocentric Activity Monitoring and Recovery , 2012, ACCV.