Deep Cooking: Predicting Relative Food Ingredient Amounts from Images

In this paper, we study the novel problem of not only predicting ingredients from a food image, but also predicting the relative amounts of the detected ingredients. We propose two prediction-based models using deep learning that output sparse and dense predictions, coupled with important semi-automatic multi-database integrative data pre-processing, to solve the problem. Experiments on a dataset of recipes collected from the Internet show the models generate encouraging experimental results.

[1]  Matthieu Guillaumin,et al.  Food-101 - Mining Discriminative Components with Random Forests , 2014, ECCV.

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Paolo Napoletano,et al.  Food Recognition: A New Dataset, Experiments, and Results , 2017, IEEE Journal of Biomedical and Health Informatics.

[4]  Amaia Salvador,et al.  Learning Cross-Modal Embeddings for Cooking Recipes and Food Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[6]  Chong-Wah Ngo,et al.  Deep Understanding of Cooking Procedure for Cross-modal Recipe Retrieval , 2018, ACM Multimedia.

[7]  Amaia Salvador,et al.  Inverse Cooking: Recipe Generation From Food Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jianhua Li,et al.  Computer vision-based food calorie estimation: dataset, method, and experiment , 2017, ArXiv.

[9]  Neel Joshi,et al.  Menu-Match: Restaurant-Specific Food Logging from Images , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[10]  Xin Zheng,et al.  Multi-view Model Contour Matching Based Food Volume Estimation , 2018 .

[11]  Wataru Shimoda,et al.  Image-Based Estimation of Real Food Size for Accurate Food Calorie Estimation , 2019, 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[12]  Xin Chen,et al.  ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognition , 2017, ArXiv.

[13]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Marios Anthimopoulos,et al.  Two-View 3D Reconstruction for Food Volume Estimation , 2017, IEEE Transactions on Multimedia.

[15]  Antonio Torralba,et al.  Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Edward J. Delp,et al.  Single-View Food Portion Estimation: Learning Image-to-Energy Mappings Using Generative Adversarial Networks , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[17]  Steven C. H. Hoi,et al.  Learning Cross-Modal Embeddings With Adversarial Networks for Cooking Recipes and Food Images , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Nassir Navab,et al.  Relative affine structure: theory and application to 3D reconstruction from perspective views , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Touradj Ebrahimi,et al.  Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model , 2016, MADiMa @ ACM Multimedia.

[20]  Chong-Wah Ngo,et al.  Deep-based Ingredient Recognition for Cooking Recipe Retrieval , 2016, ACM Multimedia.

[21]  Matthieu Cord,et al.  Recipe recognition with large multimodal food dataset , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[22]  B. Koroušić Seljak,et al.  NutriNet: A Deep Learning Food and Drink Image Recognition System for Dietary Assessment , 2017, Nutrients.