Energy expenditure estimation using visual and inertial sensors

Deriving a person's energy expenditure accurately forms the foundation for tracking physical activity levels across many health and lifestyle monitoring tasks. In this study, the authors present a method for estimating calorific expenditure from combined visual and accelerometer sensors by way of an RGB-Depth camera and a wearable inertial sensor. The proposed individual-independent framework fuses information from both modalities which leads to improved estimates beyond the accuracy of single modality and manual metabolic equivalents of task (MET) lookup table based methods. For evaluation, the authors introduce a new dataset called SPHERE_RGBD  +  Inertial_calorie, for which visual and inertial data are simultaneously obtained with indirect calorimetry ground truth measurements based on gas exchange. Experiments show that the fusion of visual and inertial data reduces the estimation error by 8 and 18% compared with the use of visual only and inertial sensor only, respectively, and by 33% compared with a MET-based approach. The authors conclude from their results that the proposed approach is suitable for home monitoring in a controlled environment.

[1]  Ivan Laptev,et al.  On Space-Time Interest Points , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[2]  Niall Twomey,et al.  Bridging e-Health and the Internet of Things: The SPHERE Project , 2015, IEEE Intelligent Systems.

[3]  Majid Mirmehdi,et al.  Real-time Estimation of Physical Activity Intensity for Daily Living , 2016 .

[4]  Wanqing Li,et al.  Activity Recognition , 2014, Computer Vision, A Reference Guide.

[5]  Petros Maragos,et al.  Multimodal gesture recognition via multiple hypotheses rescoring , 2015, J. Mach. Learn. Res..

[6]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[7]  Majid Mirmehdi,et al.  Calorie Counter: RGB-Depth Visual Estimation of Energy Expenditure at Home , 2016, ACCV Workshops.

[8]  Matjaz Gams,et al.  Context-based ensemble method for human energy expenditure estimation , 2015, Appl. Soft Comput..

[9]  Thomas Mensink,et al.  Improving the Fisher Kernel for Large-Scale Image Classification , 2010, ECCV.

[10]  Jake K. Aggarwal,et al.  Human activity recognition from 3D data: A review , 2014, Pattern Recognit. Lett..

[11]  Chao-Cheng Wu,et al.  Estimation of Calories Consumption for Aerobics Using Kinect Based Skeleton Tracking , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[12]  D. Bassett,et al.  Estimating energy expenditure using accelerometers , 2006, European Journal of Applied Physiology.

[13]  B E Ainsworth,et al.  Compendium of physical activities: an update of activity codes and MET intensities. , 2000, Medicine and science in sports and exercise.

[14]  R. C. Lummel,et al.  Estimating Activity‐related Energy Expenditure Under Sedentary Conditions Using a Tri‐axial Seismic Accelerometer , 2009, Obesity.

[15]  P. Freedson,et al.  Validity of accelerometry for the assessment of moderate intensity physical activity in the field. , 2000, Medicine and science in sports and exercise.

[16]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[17]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[18]  Ennio Gambi,et al.  Proposal and Experimental Evaluation of Fall Detection Solution Based on Wearable and Depth Data Fusion , 2015, ICT Innovations.

[19]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[20]  Thomas G. Dietterich.  Machine Learning for Sequential Data: A Review , 2002, SSPR/SPR.

[21]  Frank Vahid,et al.  Estimating Daily Energy Expenditure from Video for Assistive Monitoring , 2013, 2013 IEEE International Conference on Healthcare Informatics.

[22]  Martin Kampel,et al.  Computer vision for active and assisted living , 2016 .

[23]  Majid Mirmehdi,et al.  A comparative study of pose representation and dynamics modelling for online motion quality assessment , 2016, Comput. Vis. Image Underst..

[24]  Michael Bosse,et al.  Keyframe-based visual–inertial odometry using nonlinear optimization , 2015, Int. J. Robotics Res..

[25]  Nasser Kehtarnavaz,et al.  A Real-Time Human Action Recognition System Using Depth and Inertial Sensor Fusion , 2016, IEEE Sensors Journal.

[26]  Stephen J. McKenna,et al.  Combining embedded accelerometers with computer vision for recognizing food preparation activities , 2013, UbiComp.

[27]  Marco La Cascia,et al.  3D skeleton-based human action classification: A survey , 2016, Pattern Recognit..

[28]  Michael L. Littman,et al.  Activity Recognition from Accelerometer Data , 2005, AAAI.

[29]  Du Tran,et al.  Human Activity Recognition with Metric Learning , 2008, ECCV.

[30]  Lianyu Zheng,et al.  Real-Time Motion Tracking for Mobile Augmented/Virtual Reality Using Adaptive Visual-Inertial Fusion , 2017, Sensors.

[31]  Julien Penders,et al.  Estimating Energy Expenditure Using Body-Worn Accelerometers: A Comparison of Methods, Sensors Number and Positioning , 2015, IEEE Journal of Biomedical and Health Informatics.

[32]  Julien Penders,et al.  Estimating Oxygen Uptake During Nonsteady-State Activities and Transitions Using Wearable Sensors , 2016, IEEE Journal of Biomedical and Health Informatics.

[33]  Zicheng Liu,et al.  HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Nasser Kehtarnavaz,et al.  Improving Human Action Recognition Using Fusion of Depth Camera and Inertial Sensors , 2015, IEEE Transactions on Human-Machine Systems.

[35]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Nasser Kehtarnavaz,et al.  Fusion of Inertial and Depth Sensor Data for Robust Hand Gesture Recognition , 2014, IEEE Sensors Journal.

[37]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[38]  E. Ravussin,et al.  Determinants of 24-hour energy expenditure in man. Methods and results using a respiratory chamber. , 1986, The Journal of clinical investigation.

[39]  Yong Man Ro,et al.  Mass type-specific sparse representation for mass classification in computer-aided detection on mammograms , 2013, Biomedical engineering online.

[40]  Majid Mirmehdi,et al.  A multi-modal sensor infrastructure for healthcare in a residential environment , 2015, 2015 IEEE International Conference on Communication Workshop (ICCW).

[41]  Valérie Gay,et al.  Using mobile phones to improve medication compliance and awareness for cardiac patients , 2010, PETRA '10.

[42]  Ruzena Bajcsy,et al.  Berkeley MHAD: A comprehensive Multimodal Human Action Database , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[43]  Leo Breiman,et al.  Stacked regressions , 2004, Machine Learning.

[44]  Larry H. Matthies,et al.  Pooled motion features for first-person videos , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Guodong Guo,et al.  A survey on still image based human action recognition , 2014, Pattern Recognit..

[46]  Matthias Egger,et al.  Domains of physical activity and all-cause mortality: systematic review and dose-response meta-analysis of cohort studies. , 2011, International journal of epidemiology.

[47]  Gabriel Thomas,et al.  Human Activity Recognition using Binary Motion Image and Deep Learning , 2015, Procedia Computer Science.

[48]  Takeo Kanade,et al.  Computer Vision and Image Understanding Computer Vision for Assistive Technologies , 2022 .

[49]  Jian Cheng,et al.  Bayesian Co-Boosting for Multi-modal Gesture Recognition , 2014, Gesture Recognition.

[50]  Inmaculada Plaza,et al.  Challenges, issues and trends in fall detection systems , 2013, Biomedical engineering online.

[51]  Majid Mirmehdi,et al.  A comparative home activity monitoring study using visual and inertial sensors , 2015, 2015 17th International Conference on E-health Networking, Application & Services (HealthCom).

[52]  Nasser Kehtarnavaz,et al.  Inertial Measurement Unit-Based Wearable Computers for Assisted Living Applications: A signal processing perspective , 2016, IEEE Signal Processing Magazine.

[53]  J. D. Janssen,et al.  A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity , 1997, IEEE Transactions on Biomedical Engineering.

[54]  Guang-Zhong Yang,et al.  Deep learning for human activity recognition: A resource efficient implementation on low-power devices , 2016, 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN).