Food recognition improvement by using hyper-spectral imagery

In the past decade, dietary assessment has been one of the most popular topics of research in the food industry, which has resulted in developing several automatic or semi-automatic dietary assessment systems using visible spectrum images for food recognition. However, the main shortcoming of visible spectrum image-based systems is its inability to differentiate foods of similar color. Researchers have added additional features such as shape, size and texture to the color model to improve the overall accuracy. However, the shape and size features are rendered inefficient when recognizing food in the mixed or cooked form. The aim of this research is to show the capability of hyperspectral bands for accurate food recognition based on individual spectral bands. In this work we use a hyperspectral imaging system of 240 spectral bands with the wavelength range between 400 nm to 900 nm. The ReliefF and PCA methods select/extract less, but the most informative features which are important to learn Logistic Regression and Support Vector Machine (SVM) as binary and multiple classifiers, respectively. A total of 20 different food samples in various forms (uncut and cut), shapes, and sizes were used in this study. The prediction results indicate that the hyperspectral images have the advantages of being able to recognize different food items from a mixed form with similar color and similar food types with different colors. In our experiments the highest classification accuracy of 0.6874 with 20 different food samples is produced by SVM multiclassification of ReliefF data with the top 110 hyperspectral features. We are able to obtain approximately 0.90 accuracy, using binary classification on a specific subset of food samples.

[1]  Landu Jiang,et al.  DeepFood: Food Image Analysis and Dietary Assessment via Deep Model , 2020, IEEE Access.

[2]  Paolo Napoletano,et al.  Food Recognition: A New Dataset, Experiments, and Results , 2017, IEEE Journal of Biomedical and Health Informatics.

[3]  Tat-Seng Chua,et al.  Mixed-dish Recognition with Contextual Relation Networks , 2019, ACM Multimedia.

[4]  Igor Kononenko,et al.  ReliefF for estimation and discretization of attributes in classification, regression, and ILP probl , 1996 .

[5]  Edward J. Delp,et al.  Low Complexity Image Quality Measures for Dietary Assessment Using Mobile Devices , 2011, 2011 IEEE International Symposium on Multimedia.

[6]  J. Shan,et al.  Principal Component Analysis for Hyperspectral Image Classification , 2002 .

[7]  Da-Wen Sun,et al.  Hyperspectral imaging for food quality analysis and control , 2010 .

[8]  Yingnan Sun,et al.  Point2Volume: A Vision-Based Dietary Assessment Approach Using View Synthesis , 2020, IEEE Transactions on Industrial Informatics.

[9]  Tao Wu,et al.  Hyperspectral band selection for soybean classification based on information measure in FRS theory , 2019 .

[10]  Yiming Yang,et al.  A re-examination of text categorization methods , 1999, SIGIR '99.

[11]  Jianlong Fu,et al.  Food and Ingredient Joint Learning for Fine-Grained Recognition , 2021, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Ling-Yu Duan,et al.  From Market to Dish: Multi-ingredient Image Recognition for Personalized Recipe Recommendation , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[13]  Zhenjie Xiong,et al.  Use of Hyperspectral Imaging to Discriminate the Variety and Quality of Rice , 2015, Food Analytical Methods.

[14]  Filipe R. Cordeiro,et al.  MyFood: A Food Segmentation and Classification System to Aid Nutritional Monitoring , 2020, 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).

[15]  Yingnan Sun,et al.  A Novel Vision-based Approach for Dietary Assessment using Deep Learning View Synthesis , 2019, 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[17]  Mohammed Ahmed Subhi,et al.  Vision-Based Approaches for Automatic Food Recognition and Dietary Assessment: A Survey , 2019, IEEE Access.

[18]  Kazem Taghva,et al.  Complex Food Recognition using Hyper-Spectral Imagery , 2020, 2020 10th Annual Computing and Communication Workshop and Conference (CCWC).

[19]  David S. Ebert,et al.  The Use of Mobile Devices in Aiding Dietary Assessment and Evaluation , 2010, IEEE Journal of Selected Topics in Signal Processing.

[20]  J. A. Gualtieri,et al.  Support vector machines for classification of hyperspectral data , 2000, IGARSS 2000. IEEE 2000 International Geoscience and Remote Sensing Symposium. Taking the Pulse of the Planet: The Role of Remote Sensing in Managing the Environment. Proceedings (Cat. No.00CH37120).

[21]  Weishan Zhang,et al.  Food Image Recognition with Convolutional Neural Networks , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).

[22]  John R. Miller,et al.  Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture , 2004 .

[23]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[24]  Gamal Elmasry,et al.  Near-infrared hyperspectral imaging for grading and classification of pork. , 2012, Meat science.

[25]  Guihua Wen,et al.  MVANet: Multi-Tasks Guided Multi-View Attention Network for Chinese Food Recognition , 2020 .

[26]  Suxia Xing,et al.  Research on the Method of Hyperspectral and Image Deep Features for Bacon Classification , 2019, 2019 Chinese Control And Decision Conference (CCDC).

[27]  David S. Ebert,et al.  An Overview of the Technology Assisted Dietary Assessment Project at Purdue University , 2010, 2010 IEEE International Symposium on Multimedia.

[28]  Reinhard Klette,et al.  Deep Spectral-spatial Features of Snapshot Hyperspectral Images for Red-meat Classification , 2018, 2018 International Conference on Image and Vision Computing New Zealand (IVCNZ).

[29]  Chu Zhang,et al.  Variety Identification of Single Rice Seed Using Hyperspectral Imaging Combined with Convolutional Neural Network , 2018 .

[30]  Adolfo Martínez Usó,et al.  Clustering-Based Hyperspectral Band Selection Using Information Measures , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Fengqing Zhu,et al.  Multi-task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation , 2020, 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[32]  Abdulsalam Yassine,et al.  Food calorie measurement using deep learning neural network , 2016, 2016 IEEE International Instrumentation and Measurement Technology Conference Proceedings.

[33]  Chong-Wah Ngo,et al.  A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition , 2020, IEEE Transactions on Image Processing.

[34]  Weishan Zhang,et al.  Food Image Recognition Using Pervasive Cloud Computing , 2013, 2013 IEEE International Conference on Green Computing and Communications and IEEE Internet of Things and IEEE Cyber, Physical and Social Computing.

[35]  Shervin Shirmohammadi,et al.  A Novel SVM Based Food Recognition Method for Calorie Measurement Applications , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[36]  Lorenzo Bruzzone,et al.  Classification of hyperspectral remote sensing images with support vector machines , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Lei Yang,et al.  PFID: Pittsburgh fast-food image dataset , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[38]  Mingui Sun,et al.  3D/2D model-to-image registration for quantitative dietary assessment , 2012, 2012 38th Annual Northeast Bioengineering Conference (NEBEC).

[39]  Zhiwei Zhu,et al.  Recognition and volume estimation of food intake using a mobile device , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[40]  Linsheng Huang,et al.  Hyperspectral imaging for accurate determination of rice variety using a deep learning network with multi-feature fusion. , 2020, Spectrochimica acta. Part A, Molecular and biomolecular spectroscopy.

[41]  Bosoon Park,et al.  Hyperspectral Imaging Technology in Food and Agriculture , 2015 .

[42]  Wei-Ta Chu,et al.  Food image description based on deep-based joint food category, ingredient, and cooking method recognition , 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[43]  Min Huang,et al.  Maize seed classification using hyperspectral image coupled with multi-linear discriminant analysis , 2019 .

[44]  Vinod Vokkarane,et al.  A New Deep Learning-Based Food Recognition System for Dietary Assessment on An Edge Computing Service Infrastructure , 2018, IEEE Transactions on Services Computing.

[45]  Robert I. Damper,et al.  Band Selection for Hyperspectral Image Classification Using Mutual Information , 2006, IEEE Geoscience and Remote Sensing Letters.

[46]  Rubaiya Hafiz,et al.  Image based drinks identification for dietary assessment , 2016, 2016 International Workshop on Computational Intelligence (IWCI).

[47]  Ashok Samal,et al.  Optical scattering with hyperspectral imaging to classify longissimus dorsi muscle based on beef tenderness using multivariate modeling. , 2013, Meat science.

[48]  Stephen Marshall,et al.  Hyperspectral imaging for food applications , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[49]  Qian Du,et al.  Similarity-Based Unsupervised Band Selection for Hyperspectral Image Analysis , 2008, IEEE Geoscience and Remote Sensing Letters.

[50]  Qian Du,et al.  A joint band prioritization and band-decorrelation approach to band selection for hyperspectral image classification , 1999, IEEE Trans. Geosci. Remote. Sens..

[51]  Jindong Tan,et al.  DietCam: Multiview Food Recognition Using a Multikernel SVM , 2016, IEEE Journal of Biomedical and Health Informatics.

[52]  Chong-Wah Ngo,et al.  Deep-based Ingredient Recognition for Cooking Recipe Retrieval , 2016, ACM Multimedia.

[53]  John R. Miller,et al.  Integrated narrow-band vegetation indices for prediction of crop chlorophyll content for application to precision agriculture , 2002 .

[54]  Stephen Marshall,et al.  Varietal Classification of Rice Seeds Using RGB and Hyperspectral Images , 2020, IEEE Access.

[55]  Edward J. Delp,et al.  Analysis of food images: Features and classification , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[56]  Jun Zhou,et al.  Hyperspectral Image Classification Based on Structured Sparse Logistic Regression and Three-Dimensional Wavelet Texture Features , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[57]  Mark R. Pickering,et al.  A New Texture Feature for Improved Food Recognition Accuracy in a Mobile Phone Based Dietary Assessment System , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[58]  Edward J. Delp,et al.  Combining global and local features for food identification in dietary assessment , 2011, 2011 18th IEEE International Conference on Image Processing.

[59]  Nguyen Thi Thanh Thuy,et al.  Fresh food recognition using feature fusion , 2014, 2014 International Conference on Advanced Technologies for Communications (ATC 2014).

[60]  Mark R. Pickering,et al.  Food Volume Estimation in a Mobile Phone Based Dietary Assessment System , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[61]  Raimund Leitner,et al.  Hyperspectral fruit and vegetable classification using convolutional neural networks , 2019, Comput. Electron. Agric..

[62]  Ren Zhang Tan,et al.  Quantized Deep Residual Convolutional Neural Network for Image-Based Dietary Assessment , 2020, IEEE Access.

[63]  Edward J. Delp,et al.  An image analysis system for dietary assessment and evaluation , 2010, 2010 IEEE International Conference on Image Processing.

[64]  Jason Weston,et al.  A user's guide to support vector machines. , 2010, Methods in molecular biology.

[65]  Yuzhen Lu,et al.  Food Image Recognition by Using Convolutional Neural Networks (CNNs) , 2016, ArXiv.

[66]  M. Nanni,et al.  Hyperspectral reflectance imaging to classify lettuce varieties by optimum selected wavelengths and linear discriminant analysis , 2020 .

[67]  Yidan Bao,et al.  Application of near-infrared hyperspectral imaging for variety identification of coated maize kernels with deep learning , 2020, Infrared physics & technology.

[68]  Jindong Tan,et al.  DietCam: Regular Shape Food Recognition with a Camera Phone , 2011, 2011 International Conference on Body Sensor Networks.

[69]  Edward J. Delp,et al.  Multiple Hypotheses Image Segmentation and Classification With Application to Dietary Assessment , 2015, IEEE Journal of Biomedical and Health Informatics.