Food Recognition: A New Dataset, Experiments, and Results

We propose a new dataset for the evaluation of food recognition algorithms that can be used in dietary monitoring applications. Each image depicts a real canteen tray with dishes and foods arranged in different ways. Each tray contains multiple instances of food classes. The dataset contains 1027 canteen trays for a total of 3616 food instances belonging to 73 food classes. The food on the tray images has been manually segmented using carefully drawn polygonal boundaries. We have benchmarked the dataset by designing an automatic tray analysis pipeline that takes a tray image as input, finds the regions of interest, and predicts for each region the corresponding food class. We have experimented with three different classification strategies using also several visual descriptors. We achieve about 79% of food and tray recognition accuracy using convolutional-neural-networks-based features. The dataset, as well as the benchmark framework, are available to the research community.

[1]  Keiji Yanai,et al.  FoodCam: A real-time food recognition system on a smartphone , 2015, Multimedia Tools and Applications.

[2]  Urs Ramer,et al.  An iterative procedure for the polygonal approximation of plane curves , 1972, Comput. Graph. Image Process..

[3]  Neel Joshi,et al.  Menu-Match: Restaurant-Specific Food Logging from Images , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[4]  Keiji Yanai,et al.  Image Recognition of 85 Food Categories by Feature Fusion , 2010, 2010 IEEE International Symposium on Multimedia.

[5]  Andrea Vedaldi,et al.  MatConvNet: Convolutional Neural Networks for MATLAB , 2014, ACM Multimedia.

[6]  Giovanni Maria Farinella,et al.  Classifying food images represented as Bag of Textons , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[7]  David S. Ebert,et al.  Personal dietary assessment using mobile devices , 2009, Electronic Imaging.

[8]  Qiang Liu,et al.  A Video Processing Approach to the Study of Obesity , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[9]  Keiji Yanai,et al.  Recognition of Multiple-Food Images by Detecting Candidate Regions , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[10]  Larry S. Davis,et al.  Exploiting local features from deep networks for image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Yiannis S. Boutalis,et al.  CEDD: Color and Edge Directivity Descriptor: A Compact Descriptor for Image Indexing and Retrieval , 2008, ICVS.

[12]  Yunqian Ma,et al.  Imbalanced Learning: Foundations, Algorithms, and Applications , 2013 .

[13]  M. Spann,et al.  Colour-based texture image classification using the complex wavelet transform , 2008, 2008 5th International Conference on Electrical Engineering, Computing Science and Automatic Control.

[14]  Ming Ouhyoung,et al.  Automatic Chinese food identification and quantity estimation , 2012, SIGGRAPH Asia Technical Briefs.

[15]  Keiji Yanai,et al.  Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation , 2014, ECCV Workshops.

[16]  Keiji Yanai,et al.  FoodCam-256: A Large-scale Real-time Mobile Food RecognitionSystem employing High-Dimensional Features and Compression of Classifier Weights , 2014, ACM Multimedia.

[17]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[18]  Matti Pietikäinen,et al.  Classification with color and texture: jointly or separately? , 2004, Pattern Recognit..

[19]  Paolo Napoletano,et al.  Evaluating color texture descriptors under large variations of controlled lighting conditions , 2015, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  David H. Douglas,et al.  ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURE , 1973 .

[21]  Edward J. Delp,et al.  Combining global and local features for food identification in dietary assessment , 2011, 2011 18th IEEE International Conference on Image Processing.

[22]  Zhiwei Zhu,et al.  Recognition and volume estimation of food intake using a mobile device , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[23]  Marios Anthimopoulos,et al.  A Food Recognition System for Diabetic Patients Based on an Optimized Bag-of-Features Model , 2014, IEEE Journal of Biomedical and Health Informatics.

[24]  Matthieu Guillaumin,et al.  Food-101 - Mining Discriminative Components with Random Forests , 2014, ECCV.

[25]  B. S. Manjunath,et al.  Unsupervised Segmentation of Color-Texture Regions in Images and Video , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Yutaka Arakawa,et al.  Food Weight Estimation using Smartphone and Cutlery , 2016, IoTofHealth@MobiSys.

[27]  Kiyoharu Aizawa,et al.  FoodLog: capture, analysis and retrieval of personal food images via web , 2009, CEA '09.

[28]  Keiji Yanai,et al.  Food image recognition with deep convolutional features , 2014, UbiComp Adjunct.

[29]  Mingui Sun,et al.  Determination of food portion size by image processing , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[30]  Giovanni Maria Farinella,et al.  A Benchmark Dataset to Study the Representation of Food Images , 2014, ECCV Workshops.

[31]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Paolo Napoletano,et al.  Combining local binary patterns and local color contrast for texture classification under varying illumination. , 2014, Journal of the Optical Society of America. A, Optics, image science, and vision.

[33]  Keiji Yanai,et al.  Real-Time Mobile Food Recognition System , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[34]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Gregory D. Abowd,et al.  Leveraging Context to Support Automated Food Recognition in Restaurants , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[36]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Jindong Tan,et al.  DietCam: Automatic dietary assessment with mobile camera phones , 2012, Pervasive Mob. Comput..

[38]  Paolo Napoletano,et al.  Intensity and color descriptors for texture classification , 2013, Electronic Imaging.

[39]  Makoto Ogawa,et al.  Food Detection and Recognition Using Convolutional Neural Network , 2014, ACM Multimedia.

[40]  Paolo Napoletano,et al.  IAT - Image Annotation Tool: Manual , 2015, ArXiv.

[41]  Ziad Ahmad,et al.  A mobile phone user interface for image-based dietary assessment , 2014, Electronic Imaging.

[42]  Shervin Shirmohammadi,et al.  A Novel SVM Based Food Recognition Method for Calorie Measurement Applications , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[43]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[44]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Wen Wu,et al.  Fast food recognition from videos of eating for calorie estimation , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[46]  Edward J. Delp,et al.  Analysis of food images: Features and classification , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[47]  Steven A. Shafer,et al.  Anatomy of a color histogram , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[48]  Wanqing Li,et al.  Food image classification using local appearance and global structural information , 2014, Neurocomputing.

[49]  Edward J. Delp,et al.  Food image analysis: Segmentation, identification and weight estimation , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[50]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[51]  Paul Southam,et al.  Theoretical and experimental comparison of different approaches for color texture classification , 2011, J. Electronic Imaging.

[52]  Paolo Napoletano,et al.  Food Recognition and Leftover Estimation for Daily Diet Monitoring , 2015, ICIAP Workshops.

[53]  Abdulsalam Yassine,et al.  Food calorie measurement using deep learning neural network , 2016, 2016 IEEE International Instrumentation and Measurement Technology Conference Proceedings.

[54]  Weishan Zhang,et al.  Food Image Recognition with Convolutional Neural Networks , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).

[55]  David S. Ebert,et al.  Volume estimation using food specific shape templates in mobile image-based dietary assessment , 2011, Electronic Imaging.

[56]  Shervin Shirmohammadi,et al.  Measuring Calorie and Nutrition From Food Image , 2014, IEEE Transactions on Instrumentation and Measurement.

[57]  Shawn D. Newsam,et al.  Bag-of-visual-words and spatial extensions for land-use classification , 2010, GIS '10.

[58]  David S. Ebert,et al.  The Use of Mobile Devices in Aiding Dietary Assessment and Evaluation , 2010, IEEE Journal of Selected Topics in Signal Processing.

[59]  Shervin Shirmohammadi,et al.  An image procesing approach for calorie intake measurement , 2012, 2012 IEEE International Symposium on Medical Measurements and Applications Proceedings.

[60]  Mei Chen,et al.  Food recognition using statistics of pairwise local features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[61]  Paolo Napoletano,et al.  Cooking Action Recognition with iVAT: An Interactive Video Annotation Tool , 2013, ICIAP.

[62]  Paolo Napoletano,et al.  An interactive tool for manual, semi-automatic and automatic video annotation , 2015, Comput. Vis. Image Underst..

[63]  Lei Yang,et al.  PFID: Pittsburgh fast-food image dataset , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[64]  Keiji Yanai,et al.  A food image recognition system with Multiple Kernel Learning , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[65]  Raimondo Schettini,et al.  Image orientation detection using LBP-based features and logistic regression , 2013, Multimedia Tools and Applications.

[66]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.