Towards Building a Food Knowledge Graph for Internet of Food

Background: The deployment of various networks (e.g., Internet of Things (IoT) and mobile networks) and databases (e.g., nutrition tables and food compositional databases) in the food system generates massive information silos due to the well-known data harmonization problem. The food knowledge graph provides a unified and standardized conceptual terminology and their relationships in a structured form and thus can transform these information silos across the whole food system to a more reusable globally digitally connected Internet of Food, enabling every stage of the food system from farm-to-fork. Scope and approach: We review the evolution of food knowledge organization, from food classification, food ontology to food knowledge graphs. We then discuss the progress in food knowledge graphs from several representative applications. We finally discuss the main challenges and future directions. Key findings and conclusions: Our comprehensive summary of current research on food knowledge graphs shows that food knowledge graphs play an important role in food-oriented applications, including food search and Question Answering (QA), personalized dietary recommendation, food analysis and visualization, food traceability, and food machinery intelligent manufacturing. Future directions for food knowledge graphs cover several fields such as multimodal food knowledge graphs and food intelligence.

[1]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[2]  Yong Rui,et al.  You Are What You Eat: Exploring Rich Recipe Information for Cross-Region Food Analysis , 2018, IEEE Transactions on Multimedia.

[3]  A Møller,et al.  LanguaL Food Description: a Learning Process , 2010, European Journal of Clinical Nutrition.

[4]  João Graça,et al.  Ontology building process: The wine domain , 2005 .

[5]  Francisco J Torres-Ruiz,et al.  In search of a consumer-focused food classification system. An experimental heuristic approach to differentiate degrees of quality. , 2018, Food research international.

[6]  Michael S. Bernstein,et al.  Visual Relationship Detection with Language Priors , 2016, ECCV.

[7]  Janusz Kacprzyk,et al.  Agri-food 4.0: A survey of the supply chains and technologies for the future agriculture , 2020, Comput. Ind..

[8]  Bernd Krieg-Brückner,et al.  Formal Modelling for Cooking Assistance , 2015, Software, Services, and Systems.

[9]  Anders Møller,et al.  Review of International Food Classification and Description , 2000 .

[10]  Somnath Banerjee,et al.  CookingQA: A Question Answering System Based on Cooking Ontology , 2016, MICAI.

[11]  Ilias Tagkopoulos,et al.  Using Word Embeddings to Learn a Better Food Ontology , 2020, Frontiers in Artificial Intelligence.

[12]  Tome Eftimov,et al.  DietHub: Dietary habits analysis through understanding the content of recipes , 2020 .

[13]  Andrew Zisserman,et al.  Learning Visual Attributes , 2007, NIPS.

[14]  Shuqiang Jiang,et al.  Multi-Scale Multi-View Deep Feature Aggregation for Food Recognition , 2020, IEEE Transactions on Image Processing.

[15]  Antonio Torralba,et al.  How to Make a Pizza: Learning a Compositional Layer-Based GAN Model , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Ramesh Jain,et al.  Food Recommendation: Framework, Existing Solutions, and Challenges , 2019, IEEE Transactions on Multimedia.

[17]  Rainer Stiefelhagen,et al.  Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information , 2020, 2020 25th International Conference on Pattern Recognition (ICPR).

[18]  Tome Eftimov,et al.  The RICHFIELDS Framework for Semantic Interoperability of Food Information Across Heterogenous Information Systems , 2018, KDIR.

[19]  Mohammed J. Zaki,et al.  RECIPTOR: An Effective Pretrained Model for Recipe Representation Learning , 2020, KDD.

[20]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[21]  Saeed Al-Bukhitan,et al.  Health, Food and User's Profile Ontologies for Personalized Information Retrieval , 2015, ANT/SEIT.

[22]  Jaewoo Kang,et al.  FlavorGraph: a large-scale food-chemical graph for generating food representations and recommending food pairings , 2021, Scientific reports.

[23]  Antonio Torralba,et al.  Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  N. F. Noy,et al.  Ontology Development 101: A Guide to Creating Your First Ontology , 2001 .

[25]  Danfei Xu,et al.  Scene Graph Generation by Iterative Message Passing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  B. De Baets,et al.  ONS: an ontology for a standardized description of interventions and observational studies in nutrition , 2018, Genes & Nutrition.

[27]  Deborah L. McGuinness,et al.  FoodKG Enabled Q&A Application , 2019, ISWC Satellites.

[28]  Hongbin Pu,et al.  Efficient extraction of deep image features using convolutional neural network (CNN) for applications in detecting and analysing complex food matrices , 2021, Trends in Food Science & Technology.

[29]  Yu Chen,et al.  Personalized Food Recommendation as Constrained Question Answering over a Large-scale Food Knowledge Graph , 2021, WSDM.

[30]  David S Wishart,et al.  FOBI: an ontology to represent food intake data and associate it with metabolomic data , 2020, Database J. Biol. Databases Curation.

[31]  Wei Zhang,et al.  Knowledge vault: a web-scale approach to probabilistic knowledge fusion , 2014, KDD.

[32]  Nazli Ikizler-Cinbis,et al.  RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes , 2018, EMNLP.

[33]  Le Song,et al.  Variational Reasoning for Question Answering with Knowledge Graph , 2017, AAAI.

[34]  C. Snae,et al.  FOODS: A Food-Oriented Ontology-Driven System , 2008, 2008 2nd IEEE International Conference on Digital Ecosystems and Technologies.

[35]  Wolfram Wöß,et al.  Towards a Definition of Knowledge Graphs , 2016, SEMANTiCS.

[36]  Hoang Long Nguyen,et al.  Knowledge graph fusion for smart systems: A Survey , 2020, Inf. Fusion.

[37]  Hala Skaf-Molli,et al.  Taaable: A Case-Based System for Personalized Cooking , 2014 .

[38]  Quratulain Rajput,et al.  Ontology-Based Personalized Dietary Recommendation For Travelers , 2015 .

[39]  Guus Schreiber,et al.  A case study in ontology library construction , 1995, Artif. Intell. Medicine.

[40]  Valentina Tamma An example of food ontology for diabetes control , 2005 .

[41]  B. Koroušić Seljak,et al.  ISO-FOOD ontology: A formal representation of the knowledge within the domain of isotopes for food science. , 2019, Food chemistry.

[42]  Tania Bailoni,et al.  HeLiS: An Ontology for Supporting Healthy Lifestyles , 2018, International Semantic Web Conference.

[43]  Heiko Paulheim,et al.  Knowledge graph refinement: A survey of approaches and evaluation methods , 2016, Semantic Web.

[44]  Philip S. Yu,et al.  A Survey on Knowledge Graphs: Representation, Acquisition, and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Li Qin,et al.  Food safety Knowledge Graph and Question Answering System , 2019, ICIT.

[46]  Deborah Bateson,et al.  INFOODS Guidelines for Describing Foods: A systematic approach to describing foods to facilitate international exchange of food composition data , 1991 .

[47]  Shuqiang Jiang,et al.  A Delicious Recipe Analysis Framework for Exploring Multi-Modal Recipes with Various Attributes , 2017, ACM Multimedia.

[48]  Deborah L. McGuinness,et al.  Identifying Ingredient Substitutions Using a Knowledge Graph of Food , 2021, Frontiers in Artificial Intelligence.

[49]  Sylvie Desprès,et al.  NutriSem: A Semantics-Driven Approach to Calculating Nutritional Value of Recipes , 2020, WorldCIST.

[50]  Shuqiang Jiang,et al.  Ingredient-Guided Cascaded Multi-Attention Network for Food Recognition , 2019, ACM Multimedia.

[51]  Tome Eftimov,et al.  FoodBase corpus: a new resource of annotated food entities , 2019, Database J. Biol. Databases Curation.

[52]  Piyaporn Tumnark,et al.  Ontology-Based Personalized Dietary Recommendation for Weightlifting , 2013 .

[53]  Tat-Seng Chua,et al.  Zero-Shot Ingredient Recognition by Multi-Relational Graph Convolutional Network , 2020, AAAI.

[54]  Minyi Guo,et al.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems , 2018, CIKM.

[55]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[56]  Pietro Perona,et al.  Vision of a Visipedia , 2010, Proceedings of the IEEE.

[57]  Silvio Peroni,et al.  FOOD: FOod in Open Data , 2016, International Semantic Web Conference.

[58]  A Trichopoulou,et al.  The DAFNE initiative: the methodology for assessing dietary patterns across Europe using household budget survey data , 2001, Public Health Nutrition.

[59]  Chunyan Miao,et al.  Structure-Aware Generation Network for Recipe Generation from Images , 2020, ECCV.

[60]  Abdulsalam Yassine,et al.  Towards an "Internet of Food": Food Ontologies for the Internet of Things , 2015, Future Internet.

[61]  Abhinav Gupta,et al.  The More You Know: Using Knowledge Graphs for Image Classification , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Ramesh C. Jain,et al.  A Survey on Food Computing , 2018, ACM Comput. Surv..

[63]  Deborah L. McGuinness,et al.  FoodKG: A Semantics-Driven Knowledge Graph for Food Recommendation , 2019, SEMWEB.

[64]  Yang Chi,et al.  Knowledge Management in Healthcare Sustainability: A Smart Healthy Diet Assistant in Traditional Chinese Medicine Culture , 2018, Sustainability.

[65]  John Mylopoulos,et al.  Ontologies for Knowledge Management: An Information Systems Perspective , 2004, Knowledge and Information Systems.

[66]  Damion M. Dooley,et al.  FoodOn: a harmonized food ontology to increase global food traceability, quality control and data integration , 2018, npj Science of Food.

[67]  Jie Lin,et al.  Object Detection Meets Knowledge Graphs , 2017, IJCAI.

[68]  Manuel Noguera,et al.  Semantic-Based Recommendation of Nutrition Diets for the Elderly from Agroalimentary Thesauri , 2013, FQAS.

[69]  J. D. Ireland,et al.  Food Classification and Description , 2016 .

[70]  Xiaojun Chen,et al.  A review: Knowledge reasoning over knowledge graph , 2020, Expert Syst. Appl..

[71]  Natasha Noy,et al.  Industry-scale Knowledge Graphs: Lessons and Challenges , 2019, ACM Queue.

[72]  Zhiyuan Liu,et al.  Knowledge Representation Learning: A Quantitative Review , 2018, ArXiv.

[73]  Alexander Chistyakov,et al.  FOODpedia: Russian Food Products as a Linked Data Dataset , 2015, ESWC.

[74]  Luka Šajn,et al.  Food object recognition using a mobile device: Evaluation of currently implemented systems , 2020 .

[75]  Duygu Çelik Ertugrul FoodWiki: a Mobile App Examines Side Effects of Food Additives Via Semantic Web , 2015, Journal of Medical Systems.

[76]  Tome Eftimov,et al.  A Survey of Named-Entity Recognition Methods for Food Information Extraction , 2020, IEEE Access.

[77]  T. Oldfield,et al.  Review of the sustainability of food systems and transition using the Internet of Food , 2018, npj Science of Food.

[78]  Luca Leone Beyond Connectivity: The Internet of Food Architecture Between Ethics and the EU Citizenry , 2017 .

[79]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[80]  Ming Gao,et al.  AgriKG: An Agricultural Knowledge Graph and Its Applications , 2019, DASFAA.

[81]  Gianluca Stringhini,et al.  Kissing Cuisines: Exploring Worldwide Culinary Habits on the Web , 2016, WWW.

[82]  Fernando Batista,et al.  Ontology construction : cooking domain , 2006 .

[83]  Yike Guo,et al.  Visualizing large knowledge graphs: A performance analysis , 2018, Future Gener. Comput. Syst..

[84]  David S. Rosenblum,et al.  MMKG: Multi-Modal Knowledge Graphs , 2019, ESWC.

[85]  Diego López-de-Ipiña,et al.  Enhancing Profile and Context Aware Relevant Food Search through Knowledge Graphs , 2018, UCAmI.