Enhancing Food Intake Tracking in Long-Term Care with Automated Food Imaging and Nutrient Intake Tracking (AFINI-T) Technology

Half of long-term care (LTC) residents are malnourished increasing hospitalization, mortality, morbidity, with lower quality of life. Current tracking methods are subjective and time consuming. This paper presents the automated food imaging and nutrient intake tracking (AFINI-T) technology designed for LTC. We propose a novel convolutional autoencoder for food classification, trained on an augmented UNIMIB2016 dataset and tested on our simulated LTC food intake dataset (12 meal scenarios; up to 15 classes each; top-1 classification accuracy: 88.9%; mean intake error: -0.4 mL±36.7 mL). Nutrient intake estimation by volume was strongly linearly correlated with nutrient estimates from mass (r2 0.92 to 0.99) with good agreement between methods (σ= −2.7 to −0.01; zero within each of the limits of agreement). The AFINI-T approach is a deep-learning powered computational nutrient sensing system that may provide a novel means for more accurately and objectively tracking LTC resident food intake to support and prevent malnutrition tracking strategies.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Javier Berrocal,et al.  Systematic Literature Review of Food-Intake Monitoring in an Aging Population † , 2019, Sensors.

[3]  David S. Ebert,et al.  Volume estimation using food specific shape templates in mobile image-based dietary assessment , 2011, Electronic Imaging.

[4]  S. Simmons,et al.  Nutritional Intake Monitoring for Nursing Home Residents: A Comparison of Staff Documentation, Direct Observation, and Photography Methods , 2000, Journal of the American Geriatrics Society.

[5]  H. Chaudhury,et al.  Prevalence of inadequate micronutrient intakes of Canadian long-term care residents , 2018, British Journal of Nutrition.

[7]  Edward J. Delp,et al.  A comparison of food portion size estimation using geometric models and depth images , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[8]  Jacques Wainer,et al.  Automatic fruit and vegetable classification from images , 2010 .

[9]  Hammad Afzal,et al.  A Framework to Estimate the Nutritional Value of Food in Real Time Using Deep Learning Techniques , 2019, IEEE Access.

[10]  Jindong Tan,et al.  DietCam: Multi-view regular shape food recognition with a camera phone , 2015, Pervasive Mob. Comput..

[11]  Fernando Alfredo Auat Cheeín,et al.  Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry , 2017, Comput. Electron. Agric..

[12]  Edward J. Delp,et al.  Food image analysis: Segmentation, identification and weight estimation , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[13]  R. N. Shebiah,et al.  Fruit Recognition using Color and Texture Features , 2010 .

[14]  Victoria Hammer Castellanos,et al.  Inherent flaws in a method of estimating meal intake commonly used in long-term-care facilities. , 2002, Journal of the American Dietetic Association.

[15]  Ankur Teredesai,et al.  Dietary intake assessment using integrated sensors and software , 2012, Electronic Imaging.

[16]  Fei Liu,et al.  Application of Deep Learning in Food: A Review. , 2019, Comprehensive reviews in food science and food safety.

[17]  Mark R. Pickering,et al.  Food Volume Estimation in a Mobile Phone Based Dietary Assessment System , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[18]  V. Vucea Modified Texture Diet and Long Term Care: A Secondary Data Analysis of Making the Most of Mealtimes (M3) Project , 2017 .

[19]  Xing Zhang,et al.  A mobile structured light system for food volume estimation , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[20]  Mohammed Ahmed Subhi,et al.  Vision-Based Approaches for Automatic Food Recognition and Dietary Assessment: A Survey , 2019, IEEE Access.

[21]  G. Jensen,et al.  Nutrition of the Elderly , 1971 .

[22]  L. Harris-Kojetin,et al.  Long-Term Care Services in the United States: 2013 Overview. , 2013, Vital & health statistics. Series 3, Analytical and epidemiological studies.

[23]  Samuel Verdú,et al.  Relationship between fermentation behavior, measured with a 3D vision Structured Light technique, and the internal structure of bread , 2015 .

[24]  Stefano Cagnoni,et al.  A New Approach to Image-Based Estimation of Food Volume , 2017, Algorithms.

[25]  K. Masaki,et al.  Prevalence and measures of nutritional compromise among nursing home patients: weight loss, low body mass index, malnutrition, and feeding dependency, a systematic review of the literature. , 2013, Journal of the American Medical Directors Association.

[26]  T. Baranowski,et al.  Adapting the eButton to the Abilities of Children for Diet Assessment. , 2016, Proceedings of Measuring Behavior 2016 : 10th International Conference on Methods and Techniques in Behavioral Research. International Conference on Methods and Techniques in Behavioral Research.

[27]  Abdulsalam Yassine,et al.  Food calorie measurement using deep learning neural network , 2016, 2016 IEEE International Instrumentation and Measurement Technology Conference Proceedings.

[28]  Hong Sun,et al.  Potato volume measurement based on RGB-D camera , 2018 .

[29]  Guang-Zhong Yang,et al.  A pilot study to determine whether using a lightweight, wearable micro-camera improves dietary assessment accuracy and offers information on macronutrients and eating rate , 2015, British Journal of Nutrition.

[30]  Gabriel Taubin,et al.  VeggieVision: a produce recognition system , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[31]  Raimondo Schettini,et al.  Semantic Food Segmentation for Automatic Dietary Monitoring , 2018, 2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE-Berlin).

[32]  Beatriz Remeseiro,et al.  Grab, Pay, and Eat: Semantic Food Detection for Smart Restaurants , 2018, IEEE Transactions on Multimedia.

[33]  Xiangjun Zou,et al.  Color-, depth-, and shape-based 3D fruit detection , 2019, Precision Agriculture.

[34]  Ming Ouhyoung,et al.  Automatic Chinese food identification and quantity estimation , 2012, SIGGRAPH Asia Technical Briefs.

[35]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Brian L. Steward,et al.  Automated crop plant detection based on the fusion of color and depth images for robotic weed control , 2019, J. Field Robotics.

[37]  Vieira Bruno,et al.  A Survey on Automated Food Monitoring and Dietary Management Systems , 2017, Journal of health & medical informatics.

[38]  C. Lafferty The role of long term care. , 1977, Journal - American Health Care Association.

[39]  L. Chambers,et al.  Comparison of visual estimation methods for regular and modified textures: real-time vs digital imaging. , 2012, Journal of the Academy of Nutrition and Dietetics.

[40]  C. J. Boushey,et al.  New mobile methods for dietary assessment: review of image-assisted and image-based dietary assessment methods , 2016, Proceedings of the Nutrition Society.

[41]  Alexander Wong,et al.  Prototyping the Automated Food Imaging and Nutrient Intake Tracking System: Modified Participatory Iterative Design Sprint , 2019, JMIR human factors.

[42]  Md.Towhid Chowdhury Vegetables detection from the glossary shop for the blind. , 2013 .

[43]  Shervin Shirmohammadi,et al.  Measuring Calorie and Nutrition From Food Image , 2014, IEEE Transactions on Instrumentation and Measurement.

[44]  J. Holden,et al.  Assessing vitamin D contents in foods and supplements: challenges and needs. , 2008, The American journal of clinical nutrition.

[45]  Edward J. Delp,et al.  Context based image analysis with application in dietary assessment and evaluation , 2018, Multimedia Tools and Applications.

[46]  Kiyoharu Aizawa,et al.  Image-based Calorie Content Estimation for Dietary Assessment , 2011, 2011 IEEE International Symposium on Multimedia.

[47]  Catriona M Steele,et al.  Prevalence and Determinants of Poor Food Intake of Residents Living in Long-Term Care. , 2017, Journal of the American Medical Directors Association.

[48]  Saraju P. Mohanty,et al.  iLog: An Intelligent Device for Automatic Food Intake Monitoring and Stress Detection in the IoMT , 2020, IEEE Transactions on Consumer Electronics.

[49]  Nur Asmiza Selamat,et al.  Automatic Food Intake Monitoring Based on Chewing Activity: A Survey , 2020, IEEE Access.

[50]  Yuzhen Lu,et al.  Using composite sinusoidal patterns in structured-illumination reflectance imaging (SIRI) for enhanced detection of apple bruise ☆ , 2017 .

[51]  Corby K. Martin,et al.  A novel method to remotely measure food intake of free-living individuals in real time: the remote food photography method. , 2008, The British journal of nutrition.

[52]  Paolo Napoletano,et al.  Food Recognition: A New Dataset, Experiments, and Results , 2017, IEEE Journal of Biomedical and Health Informatics.

[53]  Kwabena T Ofei,et al.  Validation of a novel image-weighed technique for monitoring food intake and estimation of portion size in hospital settings: a pilot study , 2018, Public Health Nutrition.

[54]  Ajay Divakaran,et al.  FoodX-251: A Dataset for Fine-grained Food Classification , 2019, ArXiv.

[55]  Keiji Yanai,et al.  An Automatic Calorie Estimation System of Food Images on a Smartphone , 2016, MADiMa @ ACM Multimedia.

[56]  Yingnan Sun,et al.  Point2Volume: A Vision-Based Dietary Assessment Approach Using View Synthesis , 2020, IEEE Transactions on Industrial Informatics.

[57]  Abdulsalam Yassine,et al.  You are what you eat: So measure what you eat! , 2016, IEEE Instrumentation & Measurement Magazine.

[58]  Yingnan Sun,et al.  Food Volume Estimation Based on Deep Learning View Synthesis from a Single Depth Map , 2018, Nutrients.

[59]  Wenyao Xu,et al.  Wearable Food Intake Monitoring Technologies: A Comprehensive Review , 2017, Comput..

[60]  Mingui Sun,et al.  Accuracy of food portion size estimation from digital pictures acquired by a chest-worn camera , 2013, Public Health Nutrition.

[61]  Keiji Yanai,et al.  FoodCam: A real-time food recognition system on a smartphone , 2015, Multimedia Tools and Applications.

[62]  Edward Sazonov,et al.  A Systematic Review of Technology-Driven Methodologies for Estimation of Energy Intake , 2019, IEEE Access.

[63]  Ken Goldberg,et al.  Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[64]  D. Giavarina Understanding Bland Altman analysis , 2015, Biochemia medica.

[65]  Edward J. Delp,et al.  Model-based food volume estimation using 3D pose , 2013, 2013 IEEE International Conference on Image Processing.

[66]  C. Nowson,et al.  Nutrient intake and plate waste from an Australian residential care facility , 2007, European Journal of Clinical Nutrition.

[67]  T. Adlam,et al.  Validation of the NANA (Novel Assessment of Nutrition and Ageing) touch screen system for use at home by older adults , 2014, Experimental Gerontology.

[68]  Edward J. Delp,et al.  Single-View Food Portion Estimation Based on Geometric Models , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[69]  S. Simmons,et al.  Feeding Assistance Needs of Long‐Stay Nursing Home Residents and Staff Time to Provide Care , 2006, Journal of the American Geriatrics Society.

[70]  Yudong Zhang,et al.  Classification of Fruits Using Computer Vision and a Multiclass Support Vector Machine , 2012, Sensors.

[71]  Abdulsalam Yassine,et al.  Using graph cut segmentation for food calorie measurement , 2014, 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[72]  Keiji Yanai,et al.  Simultaneous estimation of food categories and calories with multi-task CNN , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[73]  Donald A Williamson,et al.  Comparison of digital photography to weighed and visual estimation of portion sizes. , 2003, Journal of the American Dietetic Association.

[74]  Jangmyung Lee,et al.  In Situ 3D Segmentation of Individual Plant Leaves Using a RGB-D Camera for Agricultural Automation , 2015, Sensors.

[75]  E. Huskisson,et al.  The Influence of Micronutrients on Cognitive Function and Performance , 2007, The Journal of international medical research.

[76]  Lia Bally,et al.  Volumetric Food Quantification Using Computer Vision on a Depth-Sensing Smartphone: Preclinical Study , 2019, JMIR mHealth and uHealth.

[77]  Eugenio Ivorra,et al.  Continuous monitoring of bread dough fermentation using a 3D vision Structured Light technique , 2014 .

[78]  Wataru Shimoda,et al.  CNN-Based Food Image Segmentation Without Pixel-Wise Annotation , 2015, ICIAP Workshops.

[79]  Truls Østbye,et al.  Nutritional risk predicts quality of life in elderly community-living Canadians. , 2004, The journals of gerontology. Series A, Biological sciences and medical sciences.

[80]  Zhiwei Zhu,et al.  Recognition and volume estimation of food intake using a mobile device , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[81]  Marios Anthimopoulos,et al.  Two-View 3D Reconstruction for Food Volume Estimation , 2017, IEEE Transactions on Multimedia.

[82]  Melanie Birks,et al.  Grounded theory research: A design framework for novice researchers , 2019, SAGE open medicine.

[83]  Tome Eftimov,et al.  Evaluating missing value imputation methods for food composition databases. , 2020, Food and chemical toxicology : an international journal published for the British Industrial Biological Research Association.

[84]  Edward J. Delp,et al.  Multiple Hypotheses Image Segmentation and Classification With Application to Dietary Assessment , 2015, IEEE Journal of Biomedical and Health Informatics.

[85]  Dimos Polyzois,et al.  Deep learning-based automatic volumetric damage quantification using depth camera , 2019, Automation in Construction.

[86]  R. Lu,et al.  Structured-illumination reflectance imaging coupled with phase analysis techniques for surface profiling of apples , 2018, Journal of Food Engineering.

[87]  Karine Hamm Canada’s Food Guide , 2020 .

[88]  Yingnan Sun,et al.  Image-Based Food Classification and Volume Estimation for Dietary Assessment: A Review , 2020, IEEE Journal of Biomedical and Health Informatics.

[89]  P. Guenter,et al.  Prevalence and Economic Burden of Malnutrition Diagnosis Among Patients Presenting to United States Emergency Departments , 2019, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[90]  Shady Elbassuoni,et al.  Calories Prediction from Food Images , 2017, AAAI.

[91]  H. Keller,et al.  Prevalence of Malnutrition or Risk in Residents in Long Term Care: Comparison of Four Tools , 2019, Journal of nutrition in gerontology and geriatrics.