Snap, Eat, RepEat: A Food Recognition Engine for Dietary Logging

We present a system to assist users in dietary logging habits, which performs food recognition from pictures snapped on their phone in two different scenarios. In the first scenario, called "Food in context", we exploit the GPS information of a user to determine which restaurant they are having a meal at, therefore restricting the categories to recognize to the set of items in the menu. Such context allows us to also report precise calories information to the user about their meal, since restaurant chains tend to standardize portions and provide the dietary information of each meal. In the second scenario, called "Foods in the wild" we try to recognize a cooked meal from a picture which could be snapped anywhere. We perform extensive experiments on food recognition on both scenarios, demonstrating the feasibility of our approach at scale, on a newly introduced dataset with 105K images for 500 food categories.

[1]  Edward J. Delp,et al.  The Use of Temporal Information in Food Image Analysis , 2015, ICIAP Workshops.

[2]  Neel Joshi,et al.  Menu-Match: Restaurant-Specific Food Logging from Images , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[3]  Abdulsalam Yassine,et al.  Mobile cloud based food calorie measurement , 2014, 2014 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[4]  David J. Kriegman,et al.  Learning Concept Embeddings with Combined Human-Machine Expertise , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Giovanni Maria Farinella,et al.  A Benchmark Dataset to Study the Representation of Food Images , 2014, ECCV Workshops.

[6]  Giovanni Maria Farinella,et al.  On the Exploitation of One Class Classification to Distinguish Food Vs Non-Food Images , 2015, ICIAP Workshops.

[7]  N. Busch,et al.  Food-pics: an image database for experimental research on eating and appetite , 2014, Front. Psychol..

[8]  Gian Luca Foresti,et al.  On filter banks of texture features for mobile food classification , 2015, ICDSC.

[9]  Lei Yang,et al.  PFID: Pittsburgh fast-food image dataset , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[10]  Gregory D. Abowd,et al.  Barriers and Negative Nudges: Exploring Challenges in Food Journaling , 2015, CHI.

[11]  Ronald L. Rivest,et al.  The MD5 Message-Digest Algorithm , 1992, RFC.

[12]  Kiyoharu Aizawa,et al.  Food Balance Estimation by Using Personal Dietary Tendencies in a Multimedia Food Log , 2013, IEEE Transactions on Multimedia.

[13]  Matthieu Guillaumin,et al.  Food-101 - Mining Discriminative Components with Random Forests , 2014, ECCV.

[14]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Peter I. Corke,et al.  Modelling local deep convolutional neural network features to improve fine-grained image classification , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[16]  Jacob Scharcanski.  Bringing Vision-Based Measurements into our Daily Life: A Grand Challenge for Computer Vision Systems , 2016, Front. ICT.

[17]  Keiji Yanai,et al.  Food image recognition with deep convolutional features , 2014, UbiComp Adjunct.

[18]  Gaetano Borriello,et al.  2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops Design and Evaluation of a Food Index-based Nutrition Diary , 2022 .

[19]  Makoto Ogawa,et al.  Food Detection and Recognition Using Convolutional Neural Network , 2014, ACM Multimedia.

[20]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Edward J. Delp,et al.  Food image analysis: Segmentation, identification and weight estimation , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Keiji Yanai,et al.  Food image recognition using deep convolutional network with pre-training and fine-tuning , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[24]  Keiji Yanai,et al.  Real-Time Mobile Food Recognition System , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[25]  Gregory D. Abowd,et al.  Leveraging Context to Support Automated Food Recognition in Restaurants , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[26]  Jack Hessel,et al.  Image Representations and New Domains in Neural Image Captioning , 2015, VL@EMNLP.

[27]  Shuang Wang,et al.  Geolocalized Modeling for Dish Recognition , 2015, IEEE Transactions on Multimedia.

[28]  Chong-Wah Ngo,et al.  Deep-based Ingredient Recognition for Cooking Recipe Retrieval , 2016, ACM Multimedia.

[29]  M. Carter,et al.  Adherence to a Smartphone Application for Weight Loss Compared to Website and Paper Diary: Pilot Randomized Controlled Trial , 2013, Journal of medical Internet research.

[30]  Matthieu Cord,et al.  Recipe recognition with large multimodal food dataset , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[31]  Feng Zhou,et al.  Fine-Grained Image Classification by Exploring Bipartite-Graph Labels , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Luis Herranz,et al.  A probabilistic model for food image recognition in restaurants , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[33]  Marios Anthimopoulos,et al.  Food Recognition for Dietary Assessment Using Deep Convolutional Neural Networks , 2015, ICIAP Workshops.

[34]  Keiji Yanai,et al.  Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation , 2014, ECCV Workshops.

[35]  Keiji Yanai,et al.  FoodCam: A Real-Time Mobile Food Recognition System Employing Fisher Vector , 2014, MMM.

[36]  Keiji Yanai,et al.  FoodCam: A real-time food recognition system on a smartphone , 2015, Multimedia Tools and Applications.

[37]  Kevin Murphy,et al.  What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision , 2015, NAACL.

[38]  Yoko Yamakata,et al.  Design in Everyday Cooking: Challenges for Assisting with Menu Planning and Food Preparation , 2016, HCI.

[39]  Abdulsalam Yassine,et al.  Using distance estimation and deep learning to simplify calibration in food calorie measurement , 2015, 2015 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA).

[40]  Marios Anthimopoulos,et al.  Computer Vision-Based Carbohydrate Estimation for Type 1 Patients With Diabetes Using Smartphones , 2015, Journal of diabetes science and technology.

[41]  Behjat Siddiquie,et al.  “Snap-n-Eat” , 2015, Journal of diabetes science and technology.

[42]  Gian Luca Foresti,et al.  A Structured Committee for Food Recognition , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).