A Deep Transfer Learning Solution for Food Material Recognition Using Electronic Scales

In this article, we present a novel solution to automating the procurement of food materials by using electronic scales, which can automatically identify the food materials along weighing them. Although the CNN model is regarded as one of the most effective solutions to image recognition, the traditional techniques cannot handle the mismatch problem between the lab training data and the real world data. To solve the problem, we propose to embed a partial-and-imbalanced domain adaptation technique (tree adaptation network) in the deep learning model, which can borrow knowledge from sibling classes, to overcome the imbalance problem, and transfer knowledge from the source domain to the target domain, to fight the mismatch problem between the lab training data and the real world data. Experiments show that the proposed approach outperforms state-of-the-art algorithms. Furthermore, the proposed techniques have already been used in practice.

[1]  Antonio Torralba,et al.  Is Saki #delicious?: The Food Perception Gap on Instagram and Its Relation to Health , 2017, WWW.

[2]  Michael I. Jordan,et al.  Deep Transfer Learning with Joint Adaptation Networks , 2016, ICML.

[3]  Yun Shi,et al.  3D Convolutional Neural Networks for Crop Classification with Multi-Temporal Remote Sensing Images , 2018, Remote. Sens..

[4]  Matthieu Cord,et al.  Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings , 2018, SIGIR.

[5]  Beatriz Remeseiro,et al.  Grab, Pay, and Eat: Semantic Food Detection for Smart Restaurants , 2018, IEEE Transactions on Multimedia.

[6]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[7]  Shiqin Zhang,et al.  Fast auto-clean CNN model for online prediction of food materials , 2017, J. Parallel Distributed Comput..

[8]  Nitish Srivastava,et al.  Discriminative Transfer Learning with Tree-based Priors , 2013, NIPS.

[9]  Haibo He,et al.  A Hierarchical Deep Domain Adaptation Approach for Fault Diagnosis of Power Plant Thermal System , 2019, IEEE Transactions on Industrial Informatics.

[10]  Shuang Wang,et al.  Geolocalized Modeling for Dish Recognition , 2015, IEEE Transactions on Multimedia.

[11]  Michael I. Jordan,et al.  Conditional Adversarial Domain Adaptation , 2017, NeurIPS.

[12]  Jianmin Wang,et al.  Partial Adversarial Domain Adaptation , 2018, ECCV.

[13]  Michael I. Jordan,et al.  Unsupervised Domain Adaptation with Residual Transfer Networks , 2016, NIPS.

[14]  Christoph Trattner,et al.  The Impact of Recipe Features, Social Cues and Demographics on Estimating the Healthiness of Online Recipes , 2018, ICWSM.

[15]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[16]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[17]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.

[18]  Yuan Shi,et al.  Geodesic flow kernel for unsupervised domain adaptation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Trevor Darrell,et al.  Simultaneous Deep Transfer Across Domains and Tasks , 2015, ICCV.

[20]  Nazli Ikizler-Cinbis,et al.  RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes , 2018, EMNLP.

[21]  Yoshua Bengio,et al.  Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach , 2011, ICML.

[22]  Ghulam Muhammad,et al.  Automatic Fruit Classification Using Deep Learning for Industrial Applications , 2019, IEEE Transactions on Industrial Informatics.

[23]  Trevor Darrell,et al.  LSDA: Large Scale Detection through Adaptation , 2014, NIPS.

[24]  Yong Rui,et al.  You Are What You Eat: Exploring Rich Recipe Information for Cross-Region Food Analysis , 2018, IEEE Transactions on Multimedia.

[25]  Trevor Darrell,et al.  Adapting Visual Category Models to New Domains , 2010, ECCV.

[26]  Paolo Napoletano,et al.  Food Recognition: A New Dataset, Experiments, and Results , 2017, IEEE Journal of Biomedical and Health Informatics.

[27]  Shervin Shirmohammadi,et al.  Mobile Multi-Food Recognition Using Deep Learning , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[28]  Trevor Darrell,et al.  Adversarial Discriminative Domain Adaptation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Luis Herranz,et al.  Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration , 2017, IEEE Transactions on Multimedia.

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Huiru Zheng,et al.  Combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets , 2018, Comput. Biol. Medicine.

[32]  Amaia Salvador,et al.  Learning Cross-Modal Embeddings for Cooking Recipes and Food Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Judy Czylok A SENSE OF TOUCH , 1979, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[34]  Neil D. Lawrence,et al.  Dataset Shift in Machine Learning , 2009 .

[35]  Gian Luca Foresti,et al.  Wide-Slice Residual Networks for Food Recognition , 2016, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[36]  Eugene Agichtein,et al.  Did You Really Just Have a Heart Attack?: Towards Robust Detection of Personal Health Mentions in Social Media , 2018, WWW.

[37]  Christoph Trattner,et al.  Exploiting Food Choice Biases for Healthier Recipe Recommendation , 2017, SIGIR.

[38]  Dima Damen,et al.  Scaling Egocentric Vision: The EPIC-KITCHENS Dataset , 2018, ArXiv.

[39]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[40]  Kaoru Ota,et al.  Deep Learning for Mobile Multimedia , 2017, ACM Trans. Multim. Comput. Commun. Appl..

[41]  Ramesh C. Jain,et al.  A Survey on Food Computing , 2018, ACM Comput. Surv..

[42]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Joseph Kee-Yin Ng,et al.  Toward Low-Overhead Fingerprint-Based Indoor Localization via Transfer Learning: Design, Implementation, and Evaluation , 2018, IEEE Transactions on Industrial Informatics.

[44]  Chong-Wah Ngo,et al.  Cross-modal Recipe Retrieval with Rich Food Attributes , 2017, ACM Multimedia.