From Market to Dish: Multi-ingredient Image Recognition for Personalized Recipe Recommendation

Recognition of food ingredients enables applications on recipe recommendation for developing a healthier eating habit. Existing ingredients recognition methods largely rely on ideal images captured in a controlled environment, while ingredients are usually displayed unorderly in a complex environment in the market. We propose the multi-ingredient recognition problem in the market and develop a Spatial Regularization Network (SRN) based method to solve it by using a newly collected multiple vegetable image dataset captured in the market. We further use the recognition result to develop a recipe recommendation system to satisfy the daily nutrition requirements and individual preference of each user. Experiments show that our multi-ingredient recognition outperforms previous methods over 14% in mAP and recommendation model shows an improvement of over 23% in HR@10.

[1]  Guoxiang Zeng,et al.  Fruit and vegetables classification system using image saliency and convolutional neural network , 2017, 2017 IEEE 3rd Information Technology and Mechatronics Engineering Conference (ITOEC).

[2]  Zilei Wang,et al.  VegFru: A Domain-Specific Dataset for Fine-Grained Visual Categorization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[4]  Tat-Seng Chua,et al.  Visually-aware Collaborative Food Recommendation , 2018, ArXiv.

[5]  Anand Singh Jalal,et al.  Species and variety detection of fruits and vegetables from images , 2013, Int. J. Appl. Pattern Recognit..

[6]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Wentao Fan,et al.  Fine-Grained Recognition of Vegetable Images Based on Multi-scale Convolution Neural Network , 2018, ICIC.

[8]  Kiyoharu Aizawa,et al.  Image processing based approach to food balance analysis for personal food logging , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[9]  Liang Lin,et al.  Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition , 2017, AAAI.

[10]  Luis Herranz,et al.  Food recognition and recipe analysis: integrating visual content, context and external knowledge , 2018, ArXiv.

[11]  Yu-Chiang Frank Wang,et al.  Learning Deep Latent Spaces for Multi-Label Classification , 2017, ArXiv.

[12]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[13]  Yu-Chiang Frank Wang,et al.  Order-Free RNN with Visual Attention for Multi-Label Classification , 2017, AAAI.

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Nenghai Yu,et al.  Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Jacques Wainer,et al.  Automatic fruit and vegetable classification from images , 2010 .

[17]  Chen Li,et al.  High performance vegetable classification from images based on AlexNet deep learning model , 2018 .

[18]  Georg Waltner,et al.  Personalized Dietary Self-Management Using Mobile Vision-Based Assistance , 2017, ICIAP Workshops.

[19]  Eduardo C. Garrido-Merchán,et al.  Suggesting Cooking Recipes Through Simulation and Bayesian Optimization , 2018, IDEAL.

[20]  Keiji Yanai,et al.  Recognition of Multiple-Food Images by Detecting Candidate Regions , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[21]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).