CaloriNet: From silhouettes to calorie estimation in private environments

We propose a novel deep fusion architecture, CaloriNet, for the online estimation of energy expenditure for free living monitoring in private environments, where RGB data is discarded and replaced by silhouettes. Our fused convolutional neural network architecture is trainable end-to-end, to estimate calorie expenditure, using temporal foreground silhouettes alongside accelerometer data. The network is trained and cross-validated on a publicly available dataset, SPHERE_RGBD + Inertial_calorie. Results show state-of-the-art minimum error on the estimation of energy expenditure (calories per minute), outperforming alternative, standard and single-modal techniques.

[1]  A. Byrne,et al.  The effect of exercise on depression, anxiety and other mood states: a review. , 1993, Journal of psychosomatic research.

[2]  John Staudenmayer,et al.  An artificial neural network to estimate physical activity energy expenditure and identify physical activity type from an accelerometer. , 2009, Journal of applied physiology.

[3]  Andrea Vedaldi,et al.  Dynamic Image Networks for Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Gert R. G. Lanckriet,et al.  A random forest classifier for the prediction of energy expenditure and type of physical activity from wrist and hip accelerometers , 2014, Physiological measurement.

[5]  Stephen S. Intille,et al.  Using wearable activity type detection to improve physical activity energy expenditure estimation , 2010, UbiComp.

[6]  A. Prentice,et al.  The use of heart rate monitoring in the estimation of energy expenditure: a validation study using indirect whole-body calorimetry , 1989, British Journal of Nutrition.

[7]  Dana Kulic,et al.  Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks , 2017, ICMI.

[8]  Parashkev Nachev,et al.  Computer Methods and Programs in Biomedicine NiftyNet: a deep-learning platform for medical imaging , 2022 .

[9]  Mathieu Aubry,et al.  Crafting a multi-task CNN for viewpoint estimation , 2016, BMVC.

[10]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Prasant Mohapatra,et al.  Using Smartphone Sensors for Improving Energy Expenditure Estimation , 2015, IEEE Journal of Translational Engineering in Health and Medicine.

[12]  Majid Mirmehdi,et al.  Energy expenditure estimation using visual and inertial sensors , 2018, IET Comput. Vis..

[13]  Z. Tran,et al.  Effect of Exercise on Depression , 1990, Exercise and sport sciences reviews.

[14]  Robert A. McLaughlin,et al.  Cardiac left ventricle segmentation using convolutional neural network regression , 2016, 2016 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES).

[15]  Majid Mirmehdi,et al.  Designing a video monitoring system for AAL applications: the SPHERE case study , 2016 .

[16]  Kong Y. Chen,et al.  Minimal changes in environmental temperature result in a significant increase in energy expenditure and changes in the hormonal homeostasis in healthy adults. , 2010, European journal of endocrinology.

[17]  Majid Mirmehdi,et al.  A comparative study of pose representation and dynamics modelling for online motion quality assessment , 2016, Comput. Vis. Image Underst..

[18]  Peter A. Flach,et al.  Smart homes, private homes? An empirical study of technology researchers’ perceptions of ethical issues in developing smart-home health technologies , 2017, BMC medical ethics.

[19]  J F Sallis,et al.  Compendium of physical activities: classification of energy costs of human physical activities. , 1993, Medicine and science in sports and exercise.

[20]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Majid Mirmehdi,et al.  Calorific Expenditure Estimation Using Deep Convolutional Network Features , 2018, 2018 IEEE Winter Applications of Computer Vision Workshops (WACVW).

[22]  Julien Penders,et al.  Energy expenditure estimation using wearable sensors: a new methodology for activity-specific models , 2012, Wireless Health.

[23]  Majid Mirmehdi,et al.  A multi-modal sensor infrastructure for healthcare in a residential environment , 2015, 2015 IEEE International Conference on Communication Workshop (ICCW).

[24]  B. Ainsworth,et al.  International physical activity questionnaire: 12-country reliability and validity. , 2003, Medicine and science in sports and exercise.

[25]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Peter R. Cavanagh,et al.  Exercise and physical activity for older adults , 1998 .

[27]  A. Stand.,et al.  Exercise and physical activity for older adults , 1998 .

[28]  Yeh-Liang Hsu,et al.  A Review of Accelerometry-Based Wearable Motion Detectors for Physical Activity Monitoring , 2010, Sensors.

[29]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[30]  Peter R. Rizun,et al.  Optimal Wiener Filter for a Body Mounted Inertial Attitude Sensor , 2008, Journal of Navigation.

[31]  Martina Ziefle,et al.  Medical Technology in Smart Homes: Exploring the User's Perspective on Privacy, Intimacy and Trust , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.

[32]  René Vidal,et al.  3D Pose Regression Using Convolutional Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Gang Hua,et al.  Ordinal Regression with Multiple Output CNN for Age Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jindong Tan,et al.  DietCam: Automatic dietary assessment with mobile camera phones , 2012, Pervasive Mob. Comput..

[35]  D. Bassett.  International physical activity questionnaire: 12-country reliability and validity. , 2003, Medicine and science in sports and exercise.

[36]  Majid Mirmehdi,et al.  Calorie Counter: RGB-Depth Visual Estimation of Energy Expenditure at Home , 2016, ACCV Workshops.

[37]  B. Kirby,et al.  Peak Oxygen Uptake and Physical Activity in 11- to 16-Year-Olds , 1990 .

[38]  Anoop Gupta,et al.  Linking public spaces: technical and social issues , 2001, CHI.

[39]  Julien Penders,et al.  Estimating Energy Expenditure Using Body-Worn Accelerometers: A Comparison of Methods, Sensors Number and Positioning , 2015, IEEE Journal of Biomedical and Health Informatics.

[40]  Prasant Mohapatra,et al.  Using Deep Learning for Energy Expenditure Estimation with wearable sensors , 2015, 2015 17th International Conference on E-health Networking, Application & Services (HealthCom).