Real-time food intake classification and energy expenditure estimation on a mobile device

Assessment of food intake has a wide range of applications in public health and life-style related chronic disease management. In this paper, we propose a real-time food recognition platform combined with daily activity and energy expenditure estimation. In the proposed method, food recognition is based on hierarchical classification using multiple visual cues, supported by efficient software implementation suitable for realtime mobile device execution. A Fischer Vector representation together with a set of linear classifiers are used to categorize food intake. Daily energy expenditure estimation is achieved by using the built-in inertial motion sensors of the mobile device. The performance of the vision-based food recognition algorithm is compared to the current state-of-the-art, showing improved accuracy and high computational efficiency suitable for realtime feedback. Detailed user studies have also been performed to demonstrate the practical value of the software environment.

[1]  Giovanni Maria Farinella,et al.  Representing scenes for real-time context classification on mobile devices , 2015, Pattern Recognit..

[2]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[3]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[4]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[5]  池田 敦,et al.  第9回国際凍土学会 (Ninth International Conference on Permafrost) 報告 , 2008 .

[6]  Guang-Zhong Yang,et al.  Body sensor networks , 2006 .

[7]  Edward Sazonov,et al.  Automatic Ingestion Monitor: A Novel Wearable Device for Monitoring of Ingestive Behavior , 2014, IEEE Transactions on Biomedical Engineering.

[8]  Keiji Yanai,et al.  A food image recognition system with Multiple Kernel Learning , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[9]  Marko Heikkilä,et al.  Description of interest regions with local binary patterns , 2009, Pattern Recognit..

[10]  Jindong Liu,et al.  An Intelligent Food-Intake Monitoring System Using Wearable Sensors , 2012, 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks.

[11]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[12]  Tom Baranowski,et al.  Need for technological innovation in dietary assessment. , 2010, Journal of the American Dietetic Association.

[13]  Keiji Yanai,et al.  Multiple-food recognition considering co-occurrence employing manifold ranking , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[14]  James Reinders,et al.  Intel® threading building blocks , 2008 .

[15]  Keiji Yanai,et al.  Recognition of Multiple-Food Images by Detecting Candidate Regions , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[16]  T J Cole,et al.  Biased over- or under-reporting is characteristic of individuals whether over time or by different assessment methods. , 2001, Journal of the American Dietetic Association.

[17]  Thomas Mensink,et al.  Improving the Fisher Kernel for Large-Scale Image Classification , 2010, ECCV.

[18]  Amy F. Subar,et al.  Dietary Assessment Methodology , 2017 .

[19]  Wolf-Joachim Fischer,et al.  Food Intake Activity Detection Using a Wearable Microphone System , 2011, 2011 Seventh International Conference on Intelligent Environments.

[20]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[21]  Mei Chen,et al.  Food recognition using statistics of pairwise local features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Wanqing Li,et al.  On the Combination of Local Texture and Global Structure for Food Classification , 2010, 2010 IEEE International Symposium on Multimedia.

[23]  Keiji Yanai,et al.  FoodCam: A real-time food recognition system on a smartphone , 2015, Multimedia Tools and Applications.

[24]  Florent Perronnin,et al.  Fisher Kernels on Visual Vocabularies for Image Categorization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Giovanni Maria Farinella,et al.  Classifying food images represented as Bag of Textons , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[28]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..