An Artificial Intelligence-Based System to Assess Nutrient Intake for Hospitalised Patients

Regular monitoring of nutrient intake in hospitalised patients plays a critical role in reducing the risk of disease-related malnutrition. Although several methods to estimate nutrient intake have been developed, there is still a clear demand for a more reliable and fully automated technique, as this could improve data accuracy and reduce both the burden on participants and health costs. In this paper, we propose a novel system based on artificial intelligence (AI) to accurately estimate nutrient intake, by simply processing RGB Depth (RGB-D) image pairs captured before and after meal consumption. The system includes a novel multi-task contextual network for food segmentation, a few-shot learning-based classifier built by limited training samples for food recognition, and an algorithm for 3D surface construction. This allows sequential food segmentation, recognition, and estimation of the consumed food volume, permitting fully automatic estimation of the nutrient intake for each meal. For the development and evaluation of the system, a dedicated new database containing images and nutrient recipes of 322 meals is assembled, coupled to data annotation using innovative strategies. Experimental results demonstrate that the estimated nutrient intake is highly correlated (> 0.91) to the ground truth and shows very small mean relative errors (< 20%), outperforming existing techniques proposed for nutrient intake assessment.

[1]  J. Kandiah,et al.  Visual plate waste in hospitalized patients: length of stay and diet order. , 2006, Journal of the American Dietetic Association.

[2]  Gian Luca Foresti,et al.  Wide-Slice Residual Networks for Food Recognition , 2016, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[3]  Ransford Henry Botwey,et al.  Carbohydrate Estimation by a Mobile Phone-Based System Versus Self-Estimations of Individuals With Type 1 Diabetes Mellitus: A Comparative Study , 2016, Journal of medical Internet research.

[4]  Kiyoharu Aizawa,et al.  Personalized Classifier for Food Image Recognition , 2018, IEEE Transactions on Multimedia.

[5]  Lei Yang,et al.  PFID: Pittsburgh fast-food image dataset , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[6]  Daniel Wegmann,et al.  Prevalence of undernutrition on admission to Swiss hospitals. , 2010, Clinical nutrition.

[7]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Matthieu Guillaumin,et al.  Food-101 - Mining Discriminative Components with Random Forests , 2014, ECCV.

[9]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Giovanni Maria Farinella,et al.  A Multimedia Database for Automatic Meal Assessment Systems , 2017, ICIAP Workshops.

[11]  Marios Anthimopoulos,et al.  A Food Recognition System for Diabetic Patients Based on an Optimized Bag-of-Features Model , 2014, IEEE Journal of Biomedical and Health Informatics.

[12]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[13]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[14]  D. Giavarina Understanding Bland Altman analysis , 2015, Biochemia medica.

[15]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[16]  Yingnan Sun,et al.  Food Volume Estimation Based on Deep Learning View Synthesis from a Single Depth Map , 2018, Nutrients.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jacob Benesty,et al.  Pearson Correlation Coefficient , 2009 .

[19]  Vassilis Athitsos,et al.  Cognitive Analysis of Working Memory Load from Eeg, by a Deep Recurrent Neural Network , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Marios Anthimopoulos,et al.  Two-View 3D Reconstruction for Food Volume Estimation , 2017, IEEE Transactions on Multimedia.

[22]  Petia Radeva,et al.  Food Recognition Using Fusion of Classifiers Based on CNNs , 2017, ICIAP.

[23]  Beatriz Remeseiro,et al.  Grab, Pay, and Eat: Semantic Food Detection for Smart Restaurants , 2018, IEEE Transactions on Multimedia.

[24]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Antonio Torralba,et al.  Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Wataru Shimoda,et al.  Image-Based Estimation of Real Food Size for Accurate Food Calorie Estimation , 2019, 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[27]  Corby K. Martin,et al.  Measuring food intake with digital photography. , 2014, Journal of human nutrition and dietetics : the official journal of the British Dietetic Association.

[28]  Paolo Napoletano,et al.  Food Recognition: A New Dataset, Experiments, and Results , 2017, IEEE Journal of Biomedical and Health Informatics.

[29]  Rie Akamatsu,et al.  Criterion validity of the visual estimation method for determining patients' meal intake in a community hospital. , 2016, Clinical nutrition.

[30]  Marios Anthimopoulos,et al.  Segmentation and recognition of multi-food meal images for carbohydrate counting , 2013, 13th IEEE International Conference on BioInformatics and BioEngineering.

[31]  A. Nencioni,et al.  Validation of the photography method for nutritional intake assessment in hospitalized elderly subjects , 2016, The journal of nutrition, health & aging.

[32]  Giovanni Maria Farinella,et al.  A multi-task learning approach for meal assessment , 2018, MADiMa@IJCAI.

[33]  Dorothee Volkert,et al.  The German hospital malnutrition study. , 2006, Clinical nutrition.

[34]  Ajay Divakaran,et al.  FoodX-251: A Dataset for Fine-grained Food Classification , 2019, ArXiv.

[35]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Keiji Yanai,et al.  Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation , 2014, ECCV Workshops.

[37]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[38]  Marios Anthimopoulos,et al.  Food Image Segmentation for Dietary Assessment , 2016, MADiMa @ ACM Multimedia.

[39]  P. Williams,et al.  Plate waste in hospitals and strategies for change , 2011 .

[40]  M. Laporte,et al.  Nutrition screening for seniors in health care facilities: a survey of health professionals. , 2011, Canadian journal of dietetic practice and research : a publication of Dietitians of Canada = Revue canadienne de la pratique et de la recherche en dietetique : une publication des Dietetistes du Canada.

[41]  Ke Yan,et al.  Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks , 2019, Scientific Reports.

[42]  Amos J. Storkey,et al.  Data Augmentation Generative Adversarial Networks , 2017, ICLR 2018.

[43]  Stavroula G. Mougiakakou,et al.  An Artificial Intelligence-Based System for Nutrient Intake Assessment of Hospitalised Patients* , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[44]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[45]  L. Barker,et al.  Hospital Malnutrition: Prevalence, Identification and Impact on Patients and the Healthcare System , 2011, International journal of environmental research and public health.

[46]  Tao Xiang,et al.  Learning to Compare: Relation Network for Few-Shot Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Thomas A. Funkhouser,et al.  Dilated Residual Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Zhiwei Zhu,et al.  Recognition and volume estimation of food intake using a mobile device , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[49]  S. Sullivan,et al.  Innovations in Calculating Precise Nutrient Intake of Hospitalized Patients , 2016, Nutrients.

[50]  Dwarikanath Mahapatra,et al.  Low Dose Abdominal CT Image Reconstruction: An Unsupervised Learning Based Approach , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[51]  Kwabena T Ofei,et al.  Validation of a novel image-weighed technique for monitoring food intake and estimation of portion size in hospital settings: a pilot study , 2018, Public Health Nutrition.

[52]  Richard S. Zemel,et al.  Prototypical Networks for Few-shot Learning , 2017, NIPS.

[53]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.